\"'` and save it to a variable `myquery`:"
]
},
{
"cell_type": "code",
"execution_count": 3,
"id": "c92a98ba",
"metadata": {},
"outputs": [],
"source": [
"myquery = Q('primary_diagnosis_site = \"brain\"')"
]
},
{
"cell_type": "markdown",
"id": "6e855573",
"metadata": {
"tags": [
"Help!"
]
},
"source": [
"\n",
"\n",
" \n",
"
Where did those terms come from?
\n",
" \n",
"If you aren't sure how we knew what terms to put in our search, please refer back to the
What search terms are available? notebook. \n",
"
Getting results from database\n",
"\n",
"\n"
],
"text/plain": [
"Getting results from database\n",
"\n"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
"\n",
" Total execution time: 0\n",
" min 4.349 sec 4349 ms\n",
" \n",
"\n"
],
"text/plain": [
"\n",
" Total execution time: \u001b[1;36m0\u001b[0m\n",
" min \u001b[1;36m4.349\u001b[0m sec \u001b[1;36m4349\u001b[0m ms\n",
" \n"
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"subjectresults = myquery.subject.run()"
]
},
{
"cell_type": "markdown",
"id": "fdf7fb60",
"metadata": {},
"source": [
"We saved the output in a variable `subjectresults`, so we don't get much visible output. To see what our results are, we need to look into the variable. The simplest way is to call `subjectresults` directly:"
]
},
{
"cell_type": "code",
"execution_count": 5,
"id": "cff572d7",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"\n",
" \n",
" Offset: 0\n",
" Count: 100\n",
" Total Row Count: 3015\n",
" More pages: True\n",
" "
]
},
"execution_count": 5,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"subjectresults"
]
},
{
"cell_type": "markdown",
"id": "5186036c",
"metadata": {},
"source": [
"This tells us four parameters that describe our results:\n",
"\n",
"---\n",
"\n",
"- **Offset:** This is how many rows of information we've told the query to skip in the data, here we didn't tell it to skip anything, so the offset is zero\n",
"- **Count:** This is how many rows the current page of our results table has. To keep searches fast, we default to pages with 100 rows.\n",
"- **Total Row Count:** This is how many rows are in the full results table\n",
"- **More pages:** This is always a True or False. False means that our current page has all the available results. True means that we will see only the first 100 results in this table, and will need to page through for more.\n",
"\n",
"---\n",
"\n",
"In any results variable, you can also use `print()` to display a 'pretty print' version of the results. This color codes the display in a way that can make it easier to find the information you care about:\n"
]
},
{
"cell_type": "code",
"execution_count": 6,
"id": "53cba279",
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"\n",
" \n",
" Offset: 0\n",
" Count: 100\n",
" Total Row Count: 3015\n",
" More pages: True\n",
" \n",
"\n"
],
"text/plain": [
"\n",
" \n",
" Offset: \u001b[1;36m0\u001b[0m\n",
" Count: \u001b[1;36m100\u001b[0m\n",
" Total Row Count: \u001b[1;36m3015\u001b[0m\n",
" More pages: \u001b[3;92mTrue\u001b[0m\n",
" \n"
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"print(subjectresults)"
]
},
{
"cell_type": "markdown",
"id": "9f9d4361",
"metadata": {},
"source": [
"We have 2384 subject matches, let's look at the actual table. The easiest way to do this is by using the python function `.to_dataframe()` on our `subjectresults` variable:"
]
},
{
"cell_type": "code",
"execution_count": 7,
"id": "e97e19f4",
"metadata": {
"scrolled": true
},
"outputs": [
{
"data": {
"text/html": [
"\n",
" \n",
" \n",
" | subject_id | \n",
" subject_identifier | \n",
" species | \n",
" sex | \n",
" race | \n",
" ethnicity | \n",
" days_to_birth | \n",
" subject_associated_project | \n",
" vital_status | \n",
" days_to_death | \n",
" cause_of_death | \n",
"
\n",
" | Loading... (need help?) |
\n",
"\n",
"\n"
],
"text/plain": [
" subject_id subject_identifier \\\n",
"0 C177366 [{'system': 'PDC', 'value': 'C177366'}] \n",
"1 C3N-03003 [{'system': 'IDC', 'value': 'C3N-03003'}] \n",
"2 C48216 [{'system': 'PDC', 'value': 'C48216'}] \n",
"3 C794457 [{'system': 'PDC', 'value': 'C794457'}] \n",
"4 C82041 [{'system': 'PDC', 'value': 'C82041'}] \n",
".. ... ... \n",
"95 C881049 [{'system': 'PDC', 'value': 'C881049'}] \n",
"96 GENIE-GRCC-159a16b7 [{'system': 'GDC', 'value': 'GENIE-GRCC-159a16... \n",
"97 GENIE-MSK-P-0005515 [{'system': 'GDC', 'value': 'GENIE-MSK-P-00055... \n",
"98 QIN-BRAIN-DSC-01-33 [{'system': 'IDC', 'value': 'QIN-BRAIN-DSC-01-... \n",
"99 QIN-GBM-TR-13 [{'system': 'IDC', 'value': 'QIN-GBM-TR-13'}] \n",
"\n",
" species sex race \\\n",
"0 homo sapiens male american indian or alaska native \n",
"1 homo sapiens None None \n",
"2 homo sapiens female white \n",
"3 homo sapiens male white \n",
"4 homo sapiens male not reported \n",
".. ... ... ... \n",
"95 homo sapiens male white \n",
"96 homo sapiens female not allowed to collect \n",
"97 homo sapiens male white \n",
"98 homo sapiens None None \n",
"99 homo sapiens None None \n",
"\n",
" ethnicity days_to_birth \\\n",
"0 not hispanic or latino NaN \n",
"1 None NaN \n",
"2 not hispanic or latino NaN \n",
"3 not hispanic or latino NaN \n",
"4 not hispanic or latino NaN \n",
".. ... ... \n",
"95 not hispanic or latino NaN \n",
"96 not allowed to collect NaN \n",
"97 not hispanic or latino NaN \n",
"98 None NaN \n",
"99 None NaN \n",
"\n",
" subject_associated_project vital_status \\\n",
"0 [Proteogenomic Analysis of Pediatric Brain Can... Dead \n",
"1 [cptac_gbm] None \n",
"2 [Proteogenomic Analysis of Pediatric Brain Can... Alive \n",
"3 [Proteogenomic Analysis of Pediatric Brain Can... Alive \n",
"4 [Proteogenomic Analysis of Pediatric Brain Can... Alive \n",
".. ... ... \n",
"95 [Proteogenomic Analysis of Pediatric Brain Can... Alive \n",
"96 [GENIE-GRCC] Not Reported \n",
"97 [GENIE-MSK] Not Reported \n",
"98 [qin_brain_dsc_mri] None \n",
"99 [qin_gbm_treatment_response] None \n",
"\n",
" days_to_death cause_of_death \n",
"0 NaN Not Reported \n",
"1 NaN None \n",
"2 NaN Not Reported \n",
"3 NaN Not Reported \n",
"4 NaN Not Reported \n",
".. ... ... \n",
"95 NaN Not Reported \n",
"96 NaN None \n",
"97 NaN None \n",
"98 NaN None \n",
"99 NaN None \n",
"\n",
"[100 rows x 11 columns]"
]
},
"execution_count": 7,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"subjectresults.to_dataframe()"
]
},
{
"cell_type": "markdown",
"id": "e44a65df",
"metadata": {},
"source": [
"By default `to_dataframe()` shows us the first and last five rows for the first page of our results, so we can easily preview our data.\n",
"\n",
"Since we queried the Subject endpoint, our default results tell us Subject level information, that is, information about unique individuals: their sex, race, age, species, etc. The `id` column tells us the unique identifier for each individual. The identifier column has nested information about what study or studies a Subject participated in, and will list all of their researchsubject identifiers. \n",
"\n",
"The `to_dataframe()` function converts the results to a pandas dataframe. So if we save the dataframe to a variable, we can use any [pandas dataframe functions](https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.html) to explore it. For example, lets see whether any of the Subjects in our first 100 results are black or african american. First we'll save the results to a dataframe, then subset that dataframe to only show rows where the word \"black\" appears in the \"race\" column. \"NAs\" which are shown as \"None\" in these tables, so for our filter to work, we'll need to specifically tell it to ignore NAs:\n"
]
},
{
"cell_type": "code",
"execution_count": 8,
"id": "4975c1e5",
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"\n",
" \n",
" | \n",
" subject_id | \n",
" subject_identifier | \n",
" species | \n",
" sex | \n",
" race | \n",
" ethnicity | \n",
" days_to_birth | \n",
" subject_associated_project | \n",
" vital_status | \n",
" days_to_death | \n",
" cause_of_death | \n",
"
\n",
" | Loading... (need help?) |
\n",
"\n",
"\n"
],
"text/plain": [
" subject_id subject_identifier \\\n",
"14 TCGA-06-A5U0 [{'system': 'GDC', 'value': 'TCGA-06-A5U0'}, {... \n",
"41 TCGA-06-0876 [{'system': 'GDC', 'value': 'TCGA-06-0876'}, {... \n",
"71 TCGA-06-0649 [{'system': 'GDC', 'value': 'TCGA-06-0649'}, {... \n",
"73 TCGA-4W-AA9S [{'system': 'GDC', 'value': 'TCGA-4W-AA9S'}, {... \n",
"\n",
" species sex race ethnicity \\\n",
"14 homo sapiens female black or african american not hispanic or latino \n",
"41 homo sapiens female black or african american not hispanic or latino \n",
"71 homo sapiens female black or african american not hispanic or latino \n",
"73 homo sapiens male black or african american not hispanic or latino \n",
"\n",
" days_to_birth subject_associated_project vital_status days_to_death \\\n",
"14 NaN [TCGA-GBM, tcga_gbm] Alive NaN \n",
"41 NaN [TCGA-GBM, tcga_gbm] Alive NaN \n",
"71 NaN [TCGA-GBM, tcga_gbm] Dead NaN \n",
"73 NaN [TCGA-GBM, tcga_gbm] Alive NaN \n",
"\n",
" cause_of_death \n",
"14 None \n",
"41 None \n",
"71 None \n",
"73 None "
]
},
"execution_count": 8,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"subjectdata = subjectresults.to_dataframe()\n",
"subjectdata[subjectdata['race'].str.contains(\"black\", na=False)]"
]
},
{
"cell_type": "markdown",
"id": "d8750617",
"metadata": {},
"source": [
"There are subjects in our first hundred results that meet the criteria. If we just want to be sure that the data contains some value, this might be good enough. But often we want to search the entire set of results and not just the first page. \n",
"\n",
"We'll cover how to work with large results dataframes in the Pagination notebook. Or, learn how to get summary information from search results in the Data Summaries notebook."
]
},
{
"cell_type": "markdown",
"id": "05e52f3f",
"metadata": {},
"source": [
"---\n",
"\n",
"\n",
"\n",
"
Subject Field Definitions
\n",
"\n",
"
A patient entity captures the study-independent metadata for research subjects. Human research subjects are usually not traceable to a particular person to protect the subjects privacy.\n",
"\n",
" \n",
"
\n",
"- id: The 'logical' identifier of the entity in the system of record, e.g. a UUID. This 'id' is unique within a given system. The identified entity may have a different 'id' in a different system.\"
\n",
"- identifier: A 'business' identifier for the entity, typically as provided by an external system or authority, that persists across implementing systems (i.e. a 'logical' identifier). Uses a specialized, complex 'Identifier' data type to capture information about the source of the business identifier - or a URI expressed as a string to an existing entity.
\n",
"- identifier.system: The system or namespace that defines the identifier.
\n",
"- identifier.value: The value of the identifier, as defined by the system.
\n",
"- species: The taxonomic group (e.g. species) of the subject.
\n",
"- sex: The biologic character or quality that distinguishes male and female from one another as expressed by analysis of the person's gonadal, morphologic (internal and external), chromosomal, and hormonal characteristics.
\n",
"- race: An arbitrary classification of a taxonomic group that is a division of a species. It usually arises as a consequence of geographical isolation within a species and is characterized by shared heredity, physical attributes and behavior, and in the case of humans, by common history, nationality, or geographic distribution. The provided values are based on the categories defined by the U.S. Office of Management and Business and used by the U.S. Census Bureau.
\n",
"- ethnicity: An individual's self-described social and cultural grouping, specifically whether an individual describes themselves as Hispanic or Latino. The provided values are based on the categories defined by the U.S. Office of Management and Business and used by the U.S. Census Bureau.
\n",
"- days_to_birth: Number of days between the date used for index and the date from a person's date of birth represented as a calculated negative number of days.
\n",
"- subject_associated_project: The list of Projects associated with the Subject.
\n",
"- vital_status: Coded value indicating the state or condition of being living or deceased; also includes the case where the vital status is unknown.
\n",
"- days_to_death: Number of days between the date used for index and the date from a person's date of death represented as a calculated number of days.
\n",
"- cause_of_death: Coded value indicating the circumstance or condition that results in the death of the subject.
\n",
"
\n",
"\n",
"
Getting results from database\n",
"\n",
"\n"
],
"text/plain": [
"Getting results from database\n",
"\n"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
"\n",
" Total execution time: 0\n",
" min 7.779 sec 7779 ms\n",
" \n",
"\n"
],
"text/plain": [
"\n",
" Total execution time: \u001b[1;36m0\u001b[0m\n",
" min \u001b[1;36m7.779\u001b[0m sec \u001b[1;36m7779\u001b[0m ms\n",
" \n"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
"\n",
" \n",
" Offset: 0\n",
" Count: 100\n",
" Total Row Count: 4347\n",
" More pages: True\n",
" \n",
"\n"
],
"text/plain": [
"\n",
" \n",
" Offset: \u001b[1;36m0\u001b[0m\n",
" Count: \u001b[1;36m100\u001b[0m\n",
" Total Row Count: \u001b[1;36m4347\u001b[0m\n",
" More pages: \u001b[3;92mTrue\u001b[0m\n",
" \n"
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"researchsubjectresults = myquery.researchsubject.run()\n",
"print(researchsubjectresults)"
]
},
{
"cell_type": "markdown",
"id": "977f36fe",
"metadata": {},
"source": [
"Now we see that our 2384 subjects have 3716 researchsubjects between them, that means that some, but not all, of our subjects were participants in more than one study. Let's peek at the data:"
]
},
{
"cell_type": "code",
"execution_count": 10,
"id": "a20f7e62",
"metadata": {
"scrolled": true
},
"outputs": [
{
"data": {
"text/html": [
"\n",
" \n",
" \n",
" | researchsubject_id | \n",
" researchsubject_identifier | \n",
" member_of_research_project | \n",
" primary_diagnosis_condition | \n",
" primary_diagnosis_site | \n",
" subject_id | \n",
"
\n",
" | Loading... (need help?) |
\n",
"\n",
"\n"
],
"text/plain": [
" researchsubject_id \\\n",
"0 RIDER Neuro MRI-1205970977__rider_neuro_mri \n",
"1 PGBM-019__brain_tumor_progression \n",
"2 PGBM-002__brain_tumor_progression \n",
"3 RIDER Neuro MRI-1596825834__rider_neuro_mri \n",
"4 4dd86ebd-ef16-4b2b-9ea0-5d1d7afef257 \n",
".. ... \n",
"95 W19__ivygap \n",
"96 W2__ivygap \n",
"97 W3__ivygap \n",
"98 W53__ivygap \n",
"99 W16__ivygap \n",
"\n",
" researchsubject_identifier \\\n",
"0 [{'system': 'IDC', 'value': 'RIDER Neuro MRI-1... \n",
"1 [{'system': 'IDC', 'value': 'PGBM-019__brain_t... \n",
"2 [{'system': 'IDC', 'value': 'PGBM-002__brain_t... \n",
"3 [{'system': 'IDC', 'value': 'RIDER Neuro MRI-1... \n",
"4 [{'system': 'GDC', 'value': '4dd86ebd-ef16-4b2... \n",
".. ... \n",
"95 [{'system': 'IDC', 'value': 'W19__ivygap'}] \n",
"96 [{'system': 'IDC', 'value': 'W2__ivygap'}] \n",
"97 [{'system': 'IDC', 'value': 'W3__ivygap'}] \n",
"98 [{'system': 'IDC', 'value': 'W53__ivygap'}] \n",
"99 [{'system': 'IDC', 'value': 'W16__ivygap'}] \n",
"\n",
" member_of_research_project primary_diagnosis_condition \\\n",
"0 rider_neuro_mri None \n",
"1 brain_tumor_progression None \n",
"2 brain_tumor_progression None \n",
"3 rider_neuro_mri None \n",
"4 TCGA-DLBC Mature B-Cell Lymphomas \n",
".. ... ... \n",
"95 ivygap None \n",
"96 ivygap None \n",
"97 ivygap None \n",
"98 ivygap None \n",
"99 ivygap None \n",
"\n",
" primary_diagnosis_site subject_id \n",
"0 Brain RIDER Neuro MRI-1205970977 \n",
"1 Brain PGBM-019 \n",
"2 Brain PGBM-002 \n",
"3 Brain RIDER Neuro MRI-1596825834 \n",
"4 Brain TCGA-RQ-AAAT \n",
".. ... ... \n",
"95 Brain W19 \n",
"96 Brain W2 \n",
"97 Brain W3 \n",
"98 Brain W53 \n",
"99 Brain W16 \n",
"\n",
"[100 rows x 6 columns]"
]
},
"execution_count": 10,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"researchsubjectresults.to_dataframe()"
]
},
{
"cell_type": "markdown",
"id": "4564beee",
"metadata": {},
"source": [
"Each row from the researchsubject endpoint results tells us about a subject in a given study. Using this endpoint we can find out information like what studies fit our search criteria, and also get data that we can filter to have only subjects from multiple studies, or only subjects from single studies.\n",
"\n",
"Any given subject will have one row per study they participated in. The subject_id in the last column of this view is the same as the `id` in the first column of the Subjects endpoint results. You can use this to combine information across endpoints, which is covered near the end of the Cohort Building workflow notebook.\n",
"\n",
"---\n",
"\n",
"\n",
"\n",
"
ResearchSubject Field Definitions
\n",
"\n",
"
A research subject is the entity of interest in a specific research study or project, typically a human being or an animal, but can also be a device, group of humans or animals, or a tissue sample. Human research subjects are usually not traceable to a particular person to protect the subjects privacy. This entity plays the role of the case_id in existing data. A subject who participates in 3 studies will have 3 researchsubject IDs\n",
" \n",
"
\n",
"- id: The 'logical' identifier of the entity in the system of record, e.g. a UUID. This 'id' is unique within a given system. The identified entity may have a different 'id' in a different system. For CDA, this is case_id.
\n",
"- identifier: A 'business' identifier for the entity, typically as provided by an external system or authority, that persists across implementing systems (i.e. a 'logical' identifier). Uses a specialized, complex 'Identifier' data type to capture information about the source of the business identifier - or a URI expressed as a string to an existing entity.
\n",
"- identifier.system: The system or namespace that defines the identifier.
\n",
"- identifier.value: The value of the identifier, as defined by the system.
\n",
"- member_of_research_project: A reference to the Project(s) of which this ResearchSubject is a member.
\n",
"- primary_diagnosis_condition: The text term used to describe the type of malignant disease, as categorized by the World Health Organization's (WHO) International Classification of Diseases for Oncology (ICD-O). This attribute represents the disease that qualified the subject for inclusion on the ResearchProject.
\n",
"- primary_diagnosis_site: The text term used to describe the primary site of disease, as categorized by the World Health Organization's (WHO) International Classification of Diseases for Oncology (ICD-O). This categorization groups cases into general categories. This attribute represents the primary site of disease that qualified the subject for inclusion on the ResearchProject.
\n",
"- subject_id: The 'logical' identifier of the entity in the system of record, e.g. a UUID. This 'id' is unique within a given system. The identified entity may have a different 'id' in a different system. Can be joined to the `id` field from subject results
\n",
"
\n",
"\n",
"
Getting results from database\n",
"\n",
"\n"
],
"text/plain": [
"Getting results from database\n",
"\n"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
"\n",
" Total execution time: 0\n",
" min 4.519 sec 4519 ms\n",
" \n",
"\n"
],
"text/plain": [
"\n",
" Total execution time: \u001b[1;36m0\u001b[0m\n",
" min \u001b[1;36m4.519\u001b[0m sec \u001b[1;36m4519\u001b[0m ms\n",
" \n"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
"\n",
" \n",
" \n",
" | diagnosis_id | \n",
" diagnosis_identifier | \n",
" primary_diagnosis | \n",
" age_at_diagnosis | \n",
" morphology | \n",
" stage | \n",
" grade | \n",
" method_of_diagnosis | \n",
" subject_id | \n",
" researchsubject_id | \n",
"
\n",
" | Loading... (need help?) |
\n",
"\n",
"\n"
],
"text/plain": [
" diagnosis_id \\\n",
"0 021e5e64-6bb8-5d40-b61d-0eb41612fdf6 \n",
"1 15d04341-d00e-5a4d-a4c5-3688adca5bba \n",
"2 17ad0147-433a-5d06-be6c-847ee8ed59d7 \n",
"3 2704dc4b-32e9-5043-8d37-8106fc8dffc1 \n",
"4 3b8fc610-2848-11ec-b712-0a4e2186f121 \n",
".. ... \n",
"95 986cc414-2545-495c-bad3-95df6c5bf505 \n",
"96 9b60596a-d804-4243-a24b-e94a8bb94189 \n",
"97 ae659079-fc5b-50a0-af51-1a52f60e9de4 \n",
"98 b07e96c3-742f-5f5a-9b64-558ee2f14ec4 \n",
"99 c97e5bdd-8495-5426-bcbf-f8780e4f39cf \n",
"\n",
" diagnosis_identifier \\\n",
"0 [{'system': 'GDC', 'value': '021e5e64-6bb8-5d4... \n",
"1 [{'system': 'GDC', 'value': '15d04341-d00e-5a4... \n",
"2 [{'system': 'GDC', 'value': '17ad0147-433a-5d0... \n",
"3 [{'system': 'GDC', 'value': '2704dc4b-32e9-504... \n",
"4 [{'system': 'PDC', 'value': '3b8fc610-2848-11e... \n",
".. ... \n",
"95 [{'system': 'GDC', 'value': '986cc414-2545-495... \n",
"96 [{'system': 'GDC', 'value': '9b60596a-d804-424... \n",
"97 [{'system': 'GDC', 'value': 'ae659079-fc5b-50a... \n",
"98 [{'system': 'GDC', 'value': 'b07e96c3-742f-5f5... \n",
"99 [{'system': 'GDC', 'value': 'c97e5bdd-8495-542... \n",
"\n",
" primary_diagnosis age_at_diagnosis morphology stage \\\n",
"0 Astrocytoma, anaplastic 18977.0 9401/3 None \n",
"1 Oligodendroglioma, NOS 11595.0 9450/3 None \n",
"2 Glioblastoma 24319.0 9440/3 None \n",
"3 Glioblastoma 31267.0 9440/3 None \n",
"4 Glioblastoma 17038.0 9440/3 Not Reported \n",
".. ... ... ... ... \n",
"95 Teratoma, benign NaN 9080/0 None \n",
"96 Glioblastoma 23267.0 9440/3 None \n",
"97 Glioblastoma 23567.0 9440/3 None \n",
"98 Mixed glioma 10790.0 9382/3 None \n",
"99 Glioblastoma 21748.0 9440/3 None \n",
"\n",
" grade method_of_diagnosis subject_id \\\n",
"0 not reported None TCGA-CS-6665 \n",
"1 not reported None TCGA-DU-6401 \n",
"2 not reported None TCGA-76-4934 \n",
"3 not reported None TCGA-76-4928 \n",
"4 Not Reported C3L-01049 \n",
".. ... ... ... \n",
"95 Not Reported None GENIE-DFCI-008091 \n",
"96 Not Reported None C3L-04084 \n",
"97 not reported None TCGA-81-5910 \n",
"98 not reported None TCGA-HT-7681 \n",
"99 not reported None TCGA-08-0359 \n",
"\n",
" researchsubject_id \n",
"0 c16d9f69-26f9-4ec7-b2d5-ff9f7dacfe0f \n",
"1 97cb8701-7d63-4142-b404-006fbb3ecb0b \n",
"2 d1fcd53b-6c55-42b0-9b24-492f8286f5bc \n",
"3 dfa3ef71-7006-4c6d-81c4-e0c0c7b75c26 \n",
"4 104c11d9-2139-11ea-aee1-0e1aae319e49 \n",
".. ... \n",
"95 863f6e34-ebce-4133-87e0-108c5dc58221 \n",
"96 0345c668-0597-49a9-80b6-17cc9cc9cae7 \n",
"97 a63ce9ac-24a4-484e-8bcc-717cd7aa92df \n",
"98 fbe32b94-b3ba-4a56-aeba-4cffe8de2da4 \n",
"99 0aa5b68f-2b99-44d2-b17f-f4eb17fe891e \n",
"\n",
"[100 rows x 10 columns]"
]
},
"execution_count": 11,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"diagnosisresults = myquery.diagnosis.run()\n",
"diagnosisresults.to_dataframe()"
]
},
{
"cell_type": "markdown",
"id": "c6008331",
"metadata": {},
"source": [
"\n",
"---\n",
"\n",
"\n",
"\n",
"
Diagnosis Field Definitions
\n",
"\n",
"
A collection of characteristics that describe an abnormal condition of the body as assessed at a point in time. May be used to capture information about neoplastic and non-neoplastic conditions.\n",
"\n",
" \n",
"
\n",
" - id: The 'logical' identifier of the entity in the repository, e.g. a UUID. This 'id' is unique within a given system. The identified entity may have a different 'id' in a different system.
\n",
"- identifier: A 'business' identifier for the entity, typically as provided by an external system or authority, that persists across implementing systems (i.e. a 'logical' identifier). Uses a specialized, complex 'Identifier' data type to capture information about the source of the business identifier - or a URI expressed as a string to an existing entity.
\n",
"- identifier.system: The system or namespace that defines the identifier.
\n",
"- identifier.value: The value of the identifier, as defined by the system.
\n",
" - primary_diagnosis: The diagnosis instance that qualified a subject for inclusion on a ResearchProject.
\n",
" - age_at_diagnosis: The age in days of the individual at the time of diagnosis.
\n",
" - morphology: Code that represents the histology of the disease using the third edition of the International Classification of Diseases for Oncology, published in 2000, used principally in tumor and cancer registries for coding the site (topography) and the histology (morphology) of neoplasms.
\n",
" - stage: The extent of a cancer in the body. Staging is usually based on the size of the tumor, whether lymph nodes contain cancer, and whether the cancer has spread from the original site to other parts of the body. Different diseases may use different staging criteria, please refer to the originating data source to see what staging system is reported
\n",
" - grade: The degree of abnormality of cancer cells, a measure of differentiation, the extent to which cancer cells are similar in appearance and function to healthy cells of the same tissue type. The degree of differentiation often relates to the clinical behavior of the particular tumor. Based on the microscopic findings, tumor grade is commonly described by one of four degrees of severity. Histopathologic grade of a tumor may be used to plan treatment and estimate the future course, outcome, and overall prognosis of disease. Certain types of cancers, such as soft tissue sarcoma, primary brain tumors, lymphomas, and breast have special grading systems.
\n",
" - method_of_diagnosis: The method used to confirm the patients malignant diagnosis.
\n",
" - subject_id: An identifier for the subject. Can be joined to the `id` field from subject results
\n",
" - researchsubject_id: An identifier for the researchsubject. Can be joined to the `id` field from researchsubject results
\n",
"
\n",
"\n",
"
Getting results from database\n",
"\n",
"\n"
],
"text/plain": [
"Getting results from database\n",
"\n"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
"\n",
" Total execution time: 0\n",
" min 4.61 sec 4610 ms\n",
" \n",
"\n"
],
"text/plain": [
"\n",
" Total execution time: \u001b[1;36m0\u001b[0m\n",
" min \u001b[1;36m4.61\u001b[0m sec \u001b[1;36m4610\u001b[0m ms\n",
" \n"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
"\n",
" \n",
" \n",
" | treatment_id | \n",
" treatment_identifier | \n",
" treatment_type | \n",
" treatment_outcome | \n",
" days_to_treatment_start | \n",
" days_to_treatment_end | \n",
" therapeutic_agent | \n",
" treatment_anatomic_site | \n",
" treatment_effect | \n",
" treatment_end_reason | \n",
" number_of_cycles | \n",
" subject_id | \n",
" researchsubject_id | \n",
" diagnosis_id | \n",
"
\n",
" | Loading... (need help?) |
\n",
"\n",
"\n"
],
"text/plain": [
" treatment_id \\\n",
"0 07c67ae3-8ece-5da4-be0f-e25a60f0f283 \n",
"1 0df1d067-698c-54e8-8d31-3fce30bc50a7 \n",
"2 1815e689-66a8-5eb2-932a-ff3c7fc9a083 \n",
"3 22952a1a-208e-44f2-8bc6-93c3049a0ca8 \n",
"4 273281ca-ba99-519d-bfa2-b94efac4a2c2 \n",
".. ... \n",
"95 65181048-74e8-5421-8572-c3af40cc413d \n",
"96 7b93fd57-4573-410d-998a-898648696489 \n",
"97 81d84a22-1038-5707-8f3f-451fb6461527 \n",
"98 88f67a62-7030-5448-be4e-cde8a420b2b1 \n",
"99 89839070-3421-503b-87c6-81d12161d991 \n",
"\n",
" treatment_identifier treatment_type \\\n",
"0 [{'system': 'GDC', 'value': '07c67ae3-8ece-5da... None \n",
"1 [{'system': 'GDC', 'value': '0df1d067-698c-54e... None \n",
"2 [{'system': 'GDC', 'value': '1815e689-66a8-5eb... None \n",
"3 [{'system': 'GDC', 'value': '22952a1a-208e-44f... None \n",
"4 [{'system': 'GDC', 'value': '273281ca-ba99-519... None \n",
".. ... ... \n",
"95 [{'system': 'GDC', 'value': '65181048-74e8-542... None \n",
"96 [{'system': 'GDC', 'value': '7b93fd57-4573-410... None \n",
"97 [{'system': 'GDC', 'value': '81d84a22-1038-570... None \n",
"98 [{'system': 'GDC', 'value': '88f67a62-7030-544... None \n",
"99 [{'system': 'GDC', 'value': '89839070-3421-503... None \n",
"\n",
" treatment_outcome days_to_treatment_start days_to_treatment_end \\\n",
"0 None None None \n",
"1 None None None \n",
"2 None None None \n",
"3 None None None \n",
"4 None None None \n",
".. ... ... ... \n",
"95 None None None \n",
"96 None None None \n",
"97 None None None \n",
"98 None None None \n",
"99 None None None \n",
"\n",
" therapeutic_agent treatment_anatomic_site treatment_effect \\\n",
"0 None None None \n",
"1 None None None \n",
"2 None None None \n",
"3 None None None \n",
"4 None None None \n",
".. ... ... ... \n",
"95 None None None \n",
"96 Irinotecan None None \n",
"97 None None None \n",
"98 None None None \n",
"99 None None None \n",
"\n",
" treatment_end_reason number_of_cycles subject_id \\\n",
"0 None None TCGA-08-0518 \n",
"1 None None TCGA-12-0827 \n",
"2 None None TCGA-76-6656 \n",
"3 None None HCM-BROD-0106-C71 \n",
"4 None None TCGA-02-0325 \n",
".. ... ... ... \n",
"95 None None TCGA-06-6694 \n",
"96 None None ER-ACXF \n",
"97 None None TCGA-02-0258 \n",
"98 None None TCGA-06-1802 \n",
"99 None None TCGA-RY-A83Y \n",
"\n",
" researchsubject_id diagnosis_id \n",
"0 64fa7f09-724a-45ee-a802-09e4affe077e 85441794-e4dd-54a4-88f9-16e8c3a0a585 \n",
"1 b75845ed-a19a-4caa-b362-e62b1e538152 65ef1c26-0098-511e-b1ce-2e5df9d62f87 \n",
"2 770aa1ee-aed9-4219-900e-63523cdf312f d41e67e8-4e18-5821-ae66-9a5b14da0f7c \n",
"3 afa149e0-edb7-4f8e-9f56-6e1c0ac78287 2610ed82-d82d-4d6b-8de9-5cf01e30606a \n",
"4 f49b3e5a-6d40-4018-86ae-8396455d8878 3d599cc2-4212-5a2c-b612-c15d7ad1f86f \n",
".. ... ... \n",
"95 863d58e7-91c2-4602-98ed-c6c6e0859566 2b30dbb2-0099-508a-94b8-ee6147c5635f \n",
"96 0f72c194-c923-4779-818a-fadc9e400d69 4c6ef299-f1d5-48a1-8a72-2e6b04b2eb7b \n",
"97 70fc222d-7423-4921-9007-cdf27c7f1ff7 99f94667-035a-514a-b0fd-416946347617 \n",
"98 13d12179-3182-4f41-85a2-90fd50e51480 f76fe9e8-12c5-5109-8f9c-41479b1c3378 \n",
"99 172701cf-709b-4e8f-b2fa-e2690a9e272d f1256f3e-2f3f-5ea6-805f-6fc9be900397 \n",
"\n",
"[100 rows x 14 columns]"
]
},
"execution_count": 12,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"treatmentresults = myquery.treatment.run()\n",
"treatmentresults.to_dataframe()"
]
},
{
"cell_type": "markdown",
"id": "30804682",
"metadata": {},
"source": [
"\n",
"---\n",
"\n",
"\n",
"\n",
"
Treatment Field Definitions
\n",
"\n",
"
Medication administration or other treatment types. A single research subject may have multiple treatments for a single diagnosis, and/or different diagnoses, and different treatments, across different studies\n",
"\n",
" \n",
"
\n",
" - id: The 'logical' identifier of the entity in the repository, e.g. a UUID. This 'id' is unique within a given system. The identified entity may have a different 'id' in a different system.
\n",
"- identifier: A 'business' identifier for the entity, typically as provided by an external system or authority, that persists across implementing systems (i.e. a 'logical' identifier). Uses a specialized, complex 'Identifier' data type to capture information about the source of the business identifier - or a URI expressed as a string to an existing entity.
\n",
"- identifier.system: The system or namespace that defines the identifier.
\n",
"- identifier.value: The value of the identifier, as defined by the system.
\n",
" - treatment_type: The treatment type including medication/therapeutics or other procedures.
\n",
" - treatment_outcome: The final outcome of the treatment.
\n",
" - days_to_treatment_start: The timepoint at which the treatment started.
\n",
" - days_to_treatment_end:The timepoint at which the treatment ended.
\n",
" - therapeutic_agent: One or more therapeutic agents as part of this treatment.
\n",
" - treatment_anatomic_site: The anatomical site that the treatment targets.
\n",
" - treatment_effect:The effect of a treatment on the diagnosis or tumor.
\n",
" - treatment_end_reason:The reason the treatment ended.
\n",
" - number_of_cycles:The number of treatment cycles the subject received.
\n",
" - subject_id: An identifier for the subject. Can be joined to the `id` field from subject results
\n",
" - researchsubject_id: An identifier for the researchsubject. Can be joined to the `id` field from researchsubject results
\n",
" - researchsubject_diagnosis_id: An identifier for the diagnosis. Can be joined to the `id` field from diagnosis results
\n",
"
\n",
"\n",
"
Getting results from database\n",
"\n",
"\n"
],
"text/plain": [
"Getting results from database\n",
"\n"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
"\n",
" Total execution time: 0\n",
" min 7.896 sec 7896 ms\n",
" \n",
"\n"
],
"text/plain": [
"\n",
" Total execution time: \u001b[1;36m0\u001b[0m\n",
" min \u001b[1;36m7.896\u001b[0m sec \u001b[1;36m7896\u001b[0m ms\n",
" \n"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/plain": [
"\n",
" \n",
" Offset: 0\n",
" Count: 100\n",
" Total Row Count: 39220\n",
" More pages: True\n",
" "
]
},
"execution_count": 13,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"specimenresults = myquery.specimen.run()\n",
"specimenresults"
]
},
{
"cell_type": "markdown",
"id": "7f254547",
"metadata": {},
"source": [
"Nearly 40,000 specimens meet our search criteria! We would typically expect this number to be much larger than our number of subjects or researchsubjects. First because studies will often take more than one sample per subject, and second because any given specimen might be aliquoted out to be used in multiple tests. Since we didn't specify any further filters, our results will return all of these as separate speciments. Let's look at a few:"
]
},
{
"cell_type": "code",
"execution_count": 14,
"id": "c55b0128",
"metadata": {
"scrolled": true
},
"outputs": [
{
"data": {
"text/html": [
"\n",
" \n",
" \n",
" | specimen_id | \n",
" specimen_identifier | \n",
" specimen_associated_project | \n",
" days_to_collection | \n",
" primary_disease_type | \n",
" anatomical_site | \n",
" source_material_type | \n",
" specimen_type | \n",
" derived_from_specimen | \n",
" derived_from_subject | \n",
" subject_id | \n",
" researchsubject_id | \n",
"
\n",
" | Loading... (need help?) |
\n",
"\n",
"\n"
],
"text/plain": [
" specimen_id \\\n",
"0 0024bf91-4ba7-4e0d-9e23-cb37188324c7 \n",
"1 01b98b0e-bbfc-4e0f-934a-3a6ac9486342 \n",
"2 01ec04df-91c8-4c46-a71d-fe1abe06a221 \n",
"3 0242c021-1602-420e-a245-8ee1052fdcef \n",
"4 031453eb-67ac-416e-8350-2e35f55c7773 \n",
".. ... \n",
"95 3abeff8e-5bc2-4cb9-aa06-55bd17d41f82 \n",
"96 3b60125e-2eed-4a75-9811-b7749dbe7977 \n",
"97 3b9c4291-4b37-413e-877e-0b557e960c64 \n",
"98 3bc3a6ac-de5e-4a7b-8fea-a9cf6f4bfd18 \n",
"99 3c83ded3-3515-4071-8788-6e8bf2c2a173 \n",
"\n",
" specimen_identifier \\\n",
"0 [{'system': 'GDC', 'value': '0024bf91-4ba7-4e0... \n",
"1 [{'system': 'GDC', 'value': '01b98b0e-bbfc-4e0... \n",
"2 [{'system': 'GDC', 'value': '01ec04df-91c8-4c4... \n",
"3 [{'system': 'GDC', 'value': '0242c021-1602-420... \n",
"4 [{'system': 'GDC', 'value': '031453eb-67ac-416... \n",
".. ... \n",
"95 [{'system': 'GDC', 'value': '3abeff8e-5bc2-4cb... \n",
"96 [{'system': 'GDC', 'value': '3b60125e-2eed-4a7... \n",
"97 [{'system': 'GDC', 'value': '3b9c4291-4b37-413... \n",
"98 [{'system': 'GDC', 'value': '3bc3a6ac-de5e-4a7... \n",
"99 [{'system': 'GDC', 'value': '3c83ded3-3515-407... \n",
"\n",
" specimen_associated_project days_to_collection primary_disease_type \\\n",
"0 TCGA-LGG 78.0 Gliomas \n",
"1 TCGA-LGG NaN Gliomas \n",
"2 TCGA-LGG 219.0 Gliomas \n",
"3 TCGA-LGG 0.0 Gliomas \n",
"4 TCGA-GBM NaN Gliomas \n",
".. ... ... ... \n",
"95 TCGA-GBM NaN Gliomas \n",
"96 TCGA-LGG NaN Gliomas \n",
"97 TCGA-GBM NaN Gliomas \n",
"98 TCGA-GBM NaN Gliomas \n",
"99 TCGA-GBM NaN Gliomas \n",
"\n",
" anatomical_site source_material_type specimen_type \\\n",
"0 None Blood Derived Normal portion \n",
"1 None Primary Tumor portion \n",
"2 None Primary Tumor aliquot \n",
"3 None Primary Tumor slide \n",
"4 None Primary Tumor analyte \n",
".. ... ... ... \n",
"95 None Primary Tumor aliquot \n",
"96 None Blood Derived Normal aliquot \n",
"97 None Blood Derived Normal portion \n",
"98 None Blood Derived Normal portion \n",
"99 None Primary Tumor sample \n",
"\n",
" derived_from_specimen derived_from_subject subject_id \\\n",
"0 3407f4e4-02ef-49aa-bea0-f7c496ad7371 TCGA-S9-A7QW TCGA-S9-A7QW \n",
"1 ae23f336-e901-4e45-b139-5e105060d3af TCGA-HT-7609 TCGA-HT-7609 \n",
"2 183243f7-1294-490c-a5b8-af6bd8740e01 TCGA-HW-A5KL TCGA-HW-A5KL \n",
"3 b44d1202-4b05-545c-8f00-fdad812c28ef TCGA-HT-7689 TCGA-HT-7689 \n",
"4 0efe9cfc-89f2-4096-8c3e-c300a8efa1ee TCGA-02-2486 TCGA-02-2486 \n",
".. ... ... ... \n",
"95 0ed7f9eb-ac18-45a7-8b0c-f4912d2d1e2d TCGA-32-1970 TCGA-32-1970 \n",
"96 05b422cc-7cc1-4691-a337-ed5f751b7c10 TCGA-DU-8167 TCGA-DU-8167 \n",
"97 aa3459f6-b50c-4205-8e73-d07fc50ec0ac TCGA-27-2526 TCGA-27-2526 \n",
"98 ea567da7-cbca-4657-9762-918c513c768c TCGA-19-1791 TCGA-19-1791 \n",
"99 initial specimen TCGA-19-4065 TCGA-19-4065 \n",
"\n",
" researchsubject_id \n",
"0 765588aa-075f-477f-9c18-099d018d3a1b \n",
"1 bfbc0d1e-2a0b-4c60-8a37-91e3ce4b79ee \n",
"2 0639b753-4099-4ca4-adb9-763069a614ef \n",
"3 d58b535a-6c95-43bb-ad88-0600e6447537 \n",
"4 79a68b32-29a3-47bf-9b2b-643ec7323d73 \n",
".. ... \n",
"95 6c5154d2-af36-492f-b520-d925528824e4 \n",
"96 58c82c76-9198-4c95-80f8-c043d1cea5eb \n",
"97 69d0a566-5fbf-45f6-a240-649b06868e27 \n",
"98 6633f338-34a9-44f6-8039-0b9408d6213d \n",
"99 a262928c-e20a-4c02-8114-1227e05c43e1 \n",
"\n",
"[100 rows x 12 columns]"
]
},
"execution_count": 14,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"specimenresults.to_dataframe()"
]
},
{
"cell_type": "markdown",
"id": "5ccb0f1c",
"metadata": {},
"source": [
"\n",
"---\n",
"\n",
"\n",
"\n",
"
Specimen Field Definitions
\n",
"\n",
"
Any material taken as a sample from a biological entity (living or dead), or from a physical object or the environment. Specimens are usually collected as an example of their kind, often for use in some investigation.\n",
" A given specimen will have only a single subject ID and a single researchsubject ID\n",
" \n",
"
\n",
" - id: The 'logical' identifier of the entity in the repository, e.g. a UUID. This 'id' is unique within a given system. The identified entity may have a different 'id' in a different system.
\n",
"- identifier: A 'business' identifier for the entity, typically as provided by an external system or authority, that persists across implementing systems (i.e. a 'logical' identifier). Uses a specialized, complex 'Identifier' data type to capture information about the source of the business identifier - or a URI expressed as a string to an existing entity.
\n",
"- identifier.system: The system or namespace that defines the identifier.
\n",
"- identifier.value: The value of the identifier, as defined by the system.
\n",
" - associated_project: The Project associated with the specimen.
\n",
" - days_to_collection: The number of days from the index date to either the date a sample was collected for a specific study or project, or the date a patient underwent a procedure (e.g. surgical resection) yielding a sample that was eventually used for research.
\n",
" - primary_disease_type: The text term used to describe the type of malignant disease, as categorized by the World Health Organization's (WHO) International Classification of Diseases for Oncology (ICD-O). This attribute represents the disease that qualified the subject for inclusion on the ResearchProject.
\n",
" - anatomical_site: Per GDC Dictionary, the text term that represents the name of the primary disease site of the submitted tumor sample; recommend dropping tumor; biospecimen_anatomic_site.
\n",
" - source_material_type: The general kind of material from which the specimen was derived, indicating the physical nature of the source material.
\n",
" - specimen_type: The high-level type of the specimen, based on its how it has been derived from the original extracted sample. One of: analyte, aliquot, portion, sample, or slide.
\n",
" - derived_from_specimen: A source/parent specimen from which this one was directly derived.
\n",
" - subject_id: An identifier for the subject. Can be joined to the `id` field from subject results
\n",
" - researchsubject_id: An identifier for the researchsubject. Can be joined to the `id` field from researchsubject results
\n",
"
\n",
"\n",
"
Getting results from database\n",
"\n",
"\n"
],
"text/plain": [
"Getting results from database\n",
"\n"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
"\n",
" Total execution time: 0\n",
" min 22.472 sec 22472 ms\n",
" \n",
"\n"
],
"text/plain": [
"\n",
" Total execution time: \u001b[1;36m0\u001b[0m\n",
" min \u001b[1;36m22.472\u001b[0m sec \u001b[1;36m22472\u001b[0m ms\n",
" \n"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/plain": [
"\n",
" \n",
" Offset: 0\n",
" Count: 100\n",
" Total Row Count: 4969686\n",
" More pages: True\n",
" "
]
},
"execution_count": 15,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"myquery.subject.file.run()"
]
},
{
"cell_type": "code",
"execution_count": 16,
"id": "f184595e",
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"Getting results from database\n",
"\n",
"\n"
],
"text/plain": [
"Getting results from database\n",
"\n"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
"\n",
" Total execution time: 0\n",
" min 24.53 sec 24530 ms\n",
" \n",
"\n"
],
"text/plain": [
"\n",
" Total execution time: \u001b[1;36m0\u001b[0m\n",
" min \u001b[1;36m24.53\u001b[0m sec \u001b[1;36m24530\u001b[0m ms\n",
" \n"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/plain": [
"\n",
" \n",
" Offset: 0\n",
" Count: 100\n",
" Total Row Count: 4969666\n",
" More pages: True\n",
" "
]
},
"execution_count": 16,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"myquery.researchsubject.file.run()"
]
},
{
"cell_type": "code",
"execution_count": 17,
"id": "86c5b3ec",
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"Getting results from database\n",
"\n",
"\n"
],
"text/plain": [
"Getting results from database\n",
"\n"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
"\n",
" Total execution time: 0\n",
" min 10.649 sec 10649 ms\n",
" \n",
"\n"
],
"text/plain": [
"\n",
" Total execution time: \u001b[1;36m0\u001b[0m\n",
" min \u001b[1;36m10.649\u001b[0m sec \u001b[1;36m10649\u001b[0m ms\n",
" \n"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/plain": [
"\n",
" \n",
" Offset: 0\n",
" Count: 100\n",
" Total Row Count: 771260\n",
" More pages: True\n",
" "
]
},
"execution_count": 17,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"myquery.specimen.file.run()"
]
},
{
"cell_type": "markdown",
"id": "46fcb3e4",
"metadata": {},
"source": [
"As you might expect, searching file gives us a huge number of results. This is great if you are surveying what kind of data is available, but is less useful for getting a coherent cohort. \n",
"\n",
"A better way to get files for a specific cohort is to build a more complex query using Operators. \n",
"\n",
"Another useful way to look at high level information is to use our counts feature which returns summary information rather than the full search results. Check out the Data Summaries tutorial to try it.\n"
]
},
{
"cell_type": "markdown",
"id": "549257b1",
"metadata": {},
"source": [
"\n",
"---\n",
"\n",
"\n",
"\n",
"
File Field Definitions
\n",
"\n",
"
A file is an information-bearing electronic object that contains a physical embodiment of some information using a particular character encoding.\n",
"\n",
" \n",
"
\n",
" - id: The 'logical' identifier of the entity in the system of record, e.g. a UUID. This 'id' is unique within a given system. The identified entity may have a different 'id' in a different system.
\n",
"- identifier: A 'business' identifier for the entity, typically as provided by an external system or authority, that persists across implementing systems (i.e. a 'logical' identifier). Uses a specialized, complex 'Identifier' data type to capture information about the source of the business identifier - or a URI expressed as a string to an existing entity.
\n",
"- identifier.system: The system or namespace that defines the identifier.
\n",
"- identifier.value: The value of the identifier, as defined by the system.
\n",
" - label: Short name or abbreviation for dataset. Maps to rdfs:label.
\n",
" - data_catagory: Broad categorization of the contents of the data file.
\n",
" - data_type: Specific content type of the data file.
\n",
" - file_format: Format of the data files.
\n",
" - associated_project: A reference to the Project(s) of which this ResearchSubject is a member. The associated_project may be embedded using the ref definition or may be a reference to the id for the Project - or a URI expressed as a string to an existing entity.
\n",
" - drs_uri: A string of characters used to identify a resource on the Data Repo Service(DRS). Can be used to retrieve this specific file from a server.
\n",
" - byte_size: Size of the file in bytes. Maps to dcat:byteSize.
\n",
" - checksum: The md5 value for the file. A digit representing the sum of the correct digits in a piece of stored or transmitted digital data, against which later comparisons can be made to detect errors in the data.
\n",
" - data_modality: Data modality describes the biological nature of the information gathered as the result of an activity, independent of the technology or methods used to produce the information. Always one of \"Genomic\", \"Proteomic\", or \"Imaging\".
\n",
" - imaging_modality: An imaging modality describes the imaging equipment and/or method used to acquire certain structural or functional information about the body. These include but are not limited to computed tomography (CT) and magnetic resonance imaging (MRI). Taken from the DICOM standard.
\n",
" - dbgap_accession_number: The dbgap accession number for the project.
\n",
"
\n",
"\n",
"
Getting results from database\n",
"\n",
"\n"
],
"text/plain": [
"Getting results from database\n",
"\n"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
"\n",
" Total execution time: 0\n",
" min 16.393 sec 16393 ms\n",
" \n",
"\n"
],
"text/plain": [
"\n",
" Total execution time: \u001b[1;36m0\u001b[0m\n",
" min \u001b[1;36m16.393\u001b[0m sec \u001b[1;36m16393\u001b[0m ms\n",
" \n"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/plain": [
"\n",
" \n",
" Offset: 0\n",
" Count: 100\n",
" Total Row Count: 904\n",
" More pages: True\n",
" "
]
},
"execution_count": 18,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"myquery.mutation.run()"
]
},
{
"cell_type": "code",
"execution_count": 19,
"id": "6ce81214",
"metadata": {
"scrolled": true
},
"outputs": [
{
"data": {
"text/html": [
"Getting results from database\n",
"\n",
"\n"
],
"text/plain": [
"Getting results from database\n",
"\n"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
"\n",
" Total execution time: 0\n",
" min 4.067 sec 4067 ms\n",
" \n",
"\n"
],
"text/plain": [
"\n",
" Total execution time: \u001b[1;36m0\u001b[0m\n",
" min \u001b[1;36m4.067\u001b[0m sec \u001b[1;36m4067\u001b[0m ms\n",
" \n"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
"\n",
" \n",
" \n",
" | project_short_name | \n",
" case_barcode | \n",
" sample_barcode_tumor | \n",
" sample_barcode_normal | \n",
" aliquot_barcode_tumor | \n",
" aliquot_barcode_normal | \n",
" Hugo_Symbol | \n",
" Entrez_Gene_Id | \n",
" Center | \n",
" NCBI_Build | \n",
" Chromosome | \n",
" Start_Position | \n",
" End_Position | \n",
" Strand | \n",
" Variant_Classification | \n",
" Variant_Type | \n",
" Reference_Allele | \n",
" Tumor_Seq_Allele1 | \n",
" Tumor_Seq_Allele2 | \n",
" dbSNP_RS | \n",
" dbSNP_Val_Status | \n",
" Tumor_Validation_Allele1 | \n",
" Tumor_Validation_Allele2 | \n",
" Mutation_Status | \n",
" Validation_Method | \n",
" Sequencer | \n",
" Tumor_Sample_UUID | \n",
" Matched_Norm_Sample_UUID | \n",
" HGVSc | \n",
" HGVSp | \n",
" HGVSp_Short | \n",
" Transcript_ID | \n",
" Exon_Number | \n",
" t_depth | \n",
" t_ref_count | \n",
" t_alt_count | \n",
" n_depth | \n",
" all_effects | \n",
" Allele | \n",
" Gene | \n",
" Feature | \n",
" Feature_type | \n",
" One_Consequence | \n",
" Consequence | \n",
" cDNA_position | \n",
" CDS_position | \n",
" Protein_position | \n",
" Amino_acids | \n",
" Codons | \n",
" Existing_variation | \n",
" ALLELE_NUM | \n",
" DISTANCE | \n",
" TRANSCRIPT_STRAND | \n",
" SYMBOL | \n",
" SYMBOL_SOURCE | \n",
" HGNC_ID | \n",
" BIOTYPE | \n",
" CANONICAL | \n",
" CCDS | \n",
" ENSP | \n",
" SWISSPROT | \n",
" TREMBL | \n",
" UNIPARC | \n",
" RefSeq | \n",
" SIFT | \n",
" PolyPhen | \n",
" EXON | \n",
" INTRON | \n",
" DOMAINS | \n",
" GMAF | \n",
" AFR_MAF | \n",
" AMR_MAF | \n",
" EAS_MAF | \n",
" EUR_MAF | \n",
" SAS_MAF | \n",
" AA_MAF | \n",
" EA_MAF | \n",
" CLIN_SIG | \n",
" SOMATIC | \n",
" PUBMED | \n",
" IMPACT | \n",
" PICK | \n",
" VARIANT_CLASS | \n",
" TSL | \n",
" HGVS_OFFSET | \n",
" PHENO | \n",
" MINIMISED | \n",
" ExAC_AF | \n",
" ExAC_AF_Adj | \n",
" ExAC_AF_AFR | \n",
" ExAC_AF_AMR | \n",
" ExAC_AF_EAS | \n",
" ExAC_AF_FIN | \n",
" ExAC_AF_NFE | \n",
" ExAC_AF_OTH | \n",
" ExAC_AF_SAS | \n",
" FILTER | \n",
" CONTEXT | \n",
" src_vcf_id | \n",
" tumor_bam_uuid | \n",
" normal_bam_uuid | \n",
" case_id | \n",
" GDC_FILTER | \n",
" COSMIC | \n",
" MC3_Overlap | \n",
" GDC_Validation_Status | \n",
" callerName | \n",
" fileUUID | \n",
" fileName | \n",
"
\n",
" | Loading... (need help?) |
\n",
"\n",
"\n"
],
"text/plain": [
" project_short_name case_barcode sample_barcode_tumor \\\n",
"0 TCGA-GBM TCGA-06-5415 TCGA-06-5415-01A \n",
"1 TCGA-GBM TCGA-32-5222 TCGA-32-5222-01A \n",
"2 TCGA-GBM TCGA-41-5651 TCGA-41-5651-01A \n",
"3 TCGA-LGG TCGA-HT-8109 TCGA-HT-8109-01A \n",
"4 TCGA-LGG TCGA-HW-A5KL TCGA-HW-A5KL-01A \n",
".. ... ... ... \n",
"95 TCGA-LGG TCGA-HT-7677 TCGA-HT-7677-01A \n",
"96 TCGA-GBM TCGA-14-0787 TCGA-14-0787-01A \n",
"97 TCGA-GBM TCGA-14-1821 TCGA-14-1821-01A \n",
"98 TCGA-LGG TCGA-DU-7011 TCGA-DU-7011-01A \n",
"99 TCGA-LGG TCGA-DU-7290 TCGA-DU-7290-01A \n",
"\n",
" sample_barcode_normal aliquot_barcode_tumor \\\n",
"0 TCGA-06-5415-10A TCGA-06-5415-01A-01D-1486-08 \n",
"1 TCGA-32-5222-10A TCGA-32-5222-01A-01D-1486-08 \n",
"2 TCGA-41-5651-10A TCGA-41-5651-01A-01D-1696-08 \n",
"3 TCGA-HT-8109-10A TCGA-HT-8109-01A-11D-2395-08 \n",
"4 TCGA-HW-A5KL-10A TCGA-HW-A5KL-01A-11D-A27K-08 \n",
".. ... ... \n",
"95 TCGA-HT-7677-10A TCGA-HT-7677-01A-11D-2253-08 \n",
"96 TCGA-14-0787-10A TCGA-14-0787-01A-01D-1492-08 \n",
"97 TCGA-14-1821-10A TCGA-14-1821-01A-01W-0643-08 \n",
"98 TCGA-DU-7011-10A TCGA-DU-7011-01A-11D-2024-08 \n",
"99 TCGA-DU-7290-10A TCGA-DU-7290-01A-11D-2024-08 \n",
"\n",
" aliquot_barcode_normal Hugo_Symbol Entrez_Gene_Id Center \\\n",
"0 TCGA-06-5415-10A-01D-1486-08 RASA1 5921 BI \n",
"1 TCGA-32-5222-10A-01D-1486-08 HECW1 23072 BI \n",
"2 TCGA-41-5651-10A-01D-1696-08 CDC42EP3 10602 BI \n",
"3 TCGA-HT-8109-10A-01D-2396-08 LOXL4 84171 BI \n",
"4 TCGA-HW-A5KL-10A-01D-A27N-08 UBXN1 51035 BI \n",
".. ... ... ... ... \n",
"95 TCGA-HT-7677-10A-01D-2253-08 ARID1B 57492 BI \n",
"96 TCGA-14-0787-10A-01D-1492-08 NOP56 10528 BI \n",
"97 TCGA-14-1821-10A-01W-0644-08 TP53 7157 BI \n",
"98 TCGA-DU-7011-10A-01D-2024-08 SIRPB1 10326 BI \n",
"99 TCGA-DU-7290-10A-01D-2024-08 MXRA5 25878 BI \n",
"\n",
" NCBI_Build ... tumor_bam_uuid \\\n",
"0 GRCh38 ... b49524b5-c615-4893-895d-bc2a24688ffa \n",
"1 GRCh38 ... 44e42588-b00e-4282-9f57-74c15b5aee1a \n",
"2 GRCh38 ... 06526eec-5632-4c24-a035-274bd3118661 \n",
"3 GRCh38 ... d25043dd-2196-496a-aeeb-5adadf1adb84 \n",
"4 GRCh38 ... 3ad19e57-5d90-4624-9320-9320d1535086 \n",
".. ... ... ... \n",
"95 GRCh38 ... 359da179-14d2-4e7c-901e-864f03cdad8d \n",
"96 GRCh38 ... 211cf8f0-d56b-447e-9fd8-f2622b75e94b \n",
"97 GRCh38 ... dd7b9f39-ebd0-4873-9423-74e6a9eeeba6 \n",
"98 GRCh38 ... 265b7262-df74-4239-a5f6-77c326cf5f4e \n",
"99 GRCh38 ... cbf5d324-9fa0-4e81-a5a1-0a1d447a1f19 \n",
"\n",
" normal_bam_uuid \\\n",
"0 0a26c1a1-5900-4dac-b5ee-bf7b94fbafb2 \n",
"1 41a523dd-b452-4c4c-b46b-6461cf478fcc \n",
"2 950aea08-5e79-4aa6-b015-0f9cb6836211 \n",
"3 0e6b0fb2-46ff-4901-be54-926f4bed5f73 \n",
"4 a377e4b9-2de0-457c-87bc-1253f14a42ca \n",
".. ... \n",
"95 f29abb60-b6a6-4742-a4e8-9656c501f4b8 \n",
"96 b2250dde-b989-4e6b-a075-ea96326bd3aa \n",
"97 49b9ee4c-ab34-4ac9-8524-12cf6ca1b6cf \n",
"98 e9c5af45-ba04-4ea4-ac71-7c48fc9205f0 \n",
"99 7803ef81-54b1-40f8-8093-19d608b2c8dd \n",
"\n",
" case_id GDC_FILTER \\\n",
"0 713190ed-c6c1-4695-814b-85ca9b95e6a0 None \n",
"1 e876dd29-68b0-4bf1-83d1-488c40068a35 None \n",
"2 c184c3ca-7ad3-4202-b108-cb9fd5f5d947 None \n",
"3 c7b2833a-9fe8-49f6-b888-2d27b0516f5c None \n",
"4 0639b753-4099-4ca4-adb9-763069a614ef None \n",
".. ... ... \n",
"95 00b18ad6-ff70-41fe-8f4d-0763ac8f28b9 None \n",
"96 a2338b30-f511-4163-af3b-1e4a40ff00e1 None \n",
"97 38dd826d-06dc-4a73-b852-c3d78c9c3ff4 wga_pair \n",
"98 cb598780-9e42-4167-b487-eec90ad4f36f None \n",
"99 e00a9855-c15e-46ef-988f-25ca64ec4717 None \n",
"\n",
" COSMIC MC3_Overlap \\\n",
"0 COSM2153241;COSM2153242 True \n",
"1 COSM3412060;COSM3412061 True \n",
"2 None True \n",
"3 COSM3966822 True \n",
"4 None False \n",
".. ... ... \n",
"95 None True \n",
"96 None False \n",
"97 COSM1640840;COSM307279;COSM307280;COSM307281;C... True \n",
"98 None True \n",
"99 COSM3973603;COSM3973604 True \n",
"\n",
" GDC_Validation_Status callerName \\\n",
"0 Unknown muse|mutect|somaticsniper|varscan \n",
"1 Unknown muse|mutect|varscan \n",
"2 Unknown mutect|somaticsniper \n",
"3 Unknown muse|mutect|somaticsniper|varscan \n",
"4 Unknown muse|mutect|somaticsniper|varscan \n",
".. ... ... \n",
"95 Unknown mutect|varscan \n",
"96 Unknown varscan \n",
"97 Unknown somaticsniper|varscan \n",
"98 Unknown mutect|varscan \n",
"99 Unknown muse|mutect|somaticsniper|varscan \n",
"\n",
" fileUUID \\\n",
"0 59a84472-27d4-497c-8f37-8bc447ff9374|da904cd3-... \n",
"1 59a84472-27d4-497c-8f37-8bc447ff9374|da904cd3-... \n",
"2 da904cd3-79d7-4ae3-b6c0-e7127998b3e6|61b82eef-... \n",
"3 27330980-b14d-4d4d-bd65-5eb19c597c9c|1e0694ca-... \n",
"4 27330980-b14d-4d4d-bd65-5eb19c597c9c|1e0694ca-... \n",
".. ... \n",
"95 1e0694ca-fcde-41d3-9ae3-47cfaf527f25|7381c199-... \n",
"96 8f06ee91-d5ab-4024-98fb-b6c46f9d90a9 \n",
"97 61b82eef-f454-434f-a1ca-e9928fd4b936|8f06ee91-... \n",
"98 1e0694ca-fcde-41d3-9ae3-47cfaf527f25|7381c199-... \n",
"99 27330980-b14d-4d4d-bd65-5eb19c597c9c|1e0694ca-... \n",
"\n",
" fileName \n",
"0 TCGA.GBM.muse.59a84472-27d4-497c-8f37-8bc447ff... \n",
"1 TCGA.GBM.muse.59a84472-27d4-497c-8f37-8bc447ff... \n",
"2 TCGA.GBM.mutect.da904cd3-79d7-4ae3-b6c0-e71279... \n",
"3 TCGA.LGG.muse.27330980-b14d-4d4d-bd65-5eb19c59... \n",
"4 TCGA.LGG.muse.27330980-b14d-4d4d-bd65-5eb19c59... \n",
".. ... \n",
"95 TCGA.LGG.mutect.1e0694ca-fcde-41d3-9ae3-47cfaf... \n",
"96 TCGA.GBM.varscan.8f06ee91-d5ab-4024-98fb-b6c46... \n",
"97 TCGA.GBM.somaticsniper.61b82eef-f454-434f-a1ca... \n",
"98 TCGA.LGG.mutect.1e0694ca-fcde-41d3-9ae3-47cfaf... \n",
"99 TCGA.LGG.muse.27330980-b14d-4d4d-bd65-5eb19c59... \n",
"\n",
"[100 rows x 109 columns]"
]
},
"execution_count": 19,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"mutationresults = myquery.mutation.run()\n",
"mutationresults.to_dataframe()"
]
}
],
"metadata": {
"interpreter": {
"hash": "5c867d1980d5b66d2bfc8a5903dcee074b4b68f3917a4f27f8a310cab24e9f1b"
},
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.10.6"
},
"metadata": {
"interpreter": {
"hash": "5c867d1980d5b66d2bfc8a5903dcee074b4b68f3917a4f27f8a310cab24e9f1b"
}
}
},
"nbformat": 4,
"nbformat_minor": 5
}