"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
"2022.10.24\n",
"\n"
],
"text/plain": [
"\u001b[1;36m2022.10\u001b[0m.\u001b[1;36m24\u001b[0m\n"
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"from cdapython import Q, columns, unique_terms, query\n",
"import numpy as np\n",
"import pandas as pd\n",
"from itables import init_notebook_mode, show\n",
"init_notebook_mode(all_interactive=True)\n",
"import itables.options as opt\n",
"opt.maxBytes=0\n",
"opt.scrollX=\"200px\"\n",
"opt.scrollCollapse=True\n",
"opt.paging=True\n",
"opt.maxColumns=0\n",
"print(Q.get_version())"
]
},
{
"cell_type": "markdown",
"id": "75eef23e",
"metadata": {},
"source": [
"\n",
" \n",
" \n",
"CDA data comes from three sources:\n",
"
\n",
" \n",
"The CDA makes this data searchable in five main endpoints:\n",
"\n",
"
\n",
"- subject: A patient entity captures the study-independent metadata for research subjects. Human research subjects are usually not traceable to a particular person to protect the subjects privacy.
\n",
"- researchsubject: A research subject is the entity of interest in a specific research study or project, typically a human being or an animal, but can also be a device, group of humans or animals, or a tissue sample. Human research subjects are usually not traceable to a particular person to protect the subjects privacy. This entity plays the role of the case_id in existing data. An individual who participates in 3 studies will have 3 researchsubject IDs
\n",
"- specimen: Any material taken as a sample from a biological entity (living or dead), or from a physical object or the environment. Specimens are usually collected as an example of their kind, often for use in some investigation.
\n",
"- file: A unit of data about subjects, researchsubjects, specimens, or their associated information
\n",
"- mutation: Molecular data about specific mutations, currently limited to the TCGA-READ project from GDC.
\n",
"
\n",
"and two endpoints that offer deeper information about data in the researchsubject endpoint:\n",
"
\n",
"- diagnosis: A collection of characteristics that describe an abnormal condition of the body as assessed at a point in time. May be used to capture information about neoplastic and non-neoplastic conditions.
\n",
"- treatment: Represent medication administration or other treatment types.
\n",
"
\n",
"Any metadata field can be searched from any endpoint, the only difference between search types is what type of data is returned by default. This means that you can think of the CDA as a really, really enormous spreadsheet full of data. To search this enormous spreadsheet, you'd want select columns, and then filter rows.\n",
"
![](\"https://github.com/CancerDataAggregator/readthedocs/blob/main/docs/Examples/images/Julia.png?raw=true\")
\n",
" \n",
" \n",
" Accordingly, to see what search fields are available, Julia starts by using the command `columns`:"
]
},
{
"cell_type": "code",
"execution_count": 3,
"id": "ef0dd8e5",
"metadata": {
"scrolled": true
},
"outputs": [
{
"data": {
"text/html": [
"\n",
" \n",
" \n",
" | fieldName | \n",
" endpoint | \n",
" description | \n",
" type | \n",
" mode | \n",
"
\n",
" | Loading... (need help?) |
\n",
"\n",
"\n"
],
"text/plain": [
" fieldName endpoint \\\n",
"0 ALLELE_NUM mutation \n",
"1 fileName mutation \n",
"2 PICK mutation \n",
"3 subject_associated_project subject \n",
"4 days_to_treatment_start treatment \n",
".. ... ... \n",
"169 Exon_Number mutation \n",
"170 normal_bam_uuid mutation \n",
"171 MINIMISED mutation \n",
"172 HGVSc mutation \n",
"173 PUBMED mutation \n",
"\n",
" description type mode \n",
"0 Allele number from input; 0 is reference, 1 is... STRING NULLABLE \n",
"1 |-delimited list of name of underlying MAF file STRING NULLABLE \n",
"2 Indicates if this block of consequence data wa... STRING NULLABLE \n",
"3 The list of Projects associated with the Subject. STRING REPEATED \n",
"4 The timepoint at which the treatment started. INTEGER NULLABLE \n",
".. ... ... ... \n",
"169 The exon number (out of total number) STRING NULLABLE \n",
"170 Unique GDC identifier for the underlying norma... STRING NULLABLE \n",
"171 Alleles in this variant have been converted to... STRING NULLABLE \n",
"172 The coding sequence of the variant in HGVS rec... STRING NULLABLE \n",
"173 Pubmed ID(s) of publications that cite existin... STRING NULLABLE \n",
"\n",
"[174 rows x 5 columns]"
]
},
"execution_count": 3,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"columns().to_dataframe()"
]
},
{
"cell_type": "markdown",
"id": "bd05eba2",
"metadata": {},
"source": [
"\n",
" \n",
" \n",
"There are a lot of columns in the CDA data, but Julia is most interested in diagnosis data, so she searches the description column for \"diagnosis\":"
]
},
{
"cell_type": "code",
"execution_count": 4,
"id": "536970c4",
"metadata": {
"scrolled": true
},
"outputs": [
{
"data": {
"text/html": [
"\n",
" \n",
" | \n",
" fieldName | \n",
" endpoint | \n",
" description | \n",
" type | \n",
" mode | \n",
"
\n",
" | Loading... (need help?) |
\n",
"\n",
"\n"
],
"text/plain": [
" fieldName endpoint \\\n",
"0 age_at_diagnosis diagnosis \n",
"1 diagnosis_identifier_system diagnosis \n",
"2 primary_diagnosis_condition researchsubject \n",
"3 method_of_diagnosis diagnosis \n",
"4 diagnosis_id diagnosis \n",
"5 diagnosis_identifier_value diagnosis \n",
"6 primary_diagnosis_site researchsubject \n",
"7 primary_diagnosis diagnosis \n",
"\n",
" description type mode \n",
"0 The age in days of the individual at the time ... INTEGER NULLABLE \n",
"1 The system or namespace that defines the ident... STRING NULLABLE \n",
"2 The text term used to describe the type of mal... STRING NULLABLE \n",
"3 The method used to confirm the subjects malign... STRING NULLABLE \n",
"4 The 'logical' identifier of the entity in the ... STRING REQUIRED \n",
"5 The value of the identifier, as defined by the... STRING NULLABLE \n",
"6 The text term used to describe the primary sit... STRING NULLABLE \n",
"7 The diagnosis instance that qualified a subjec... STRING NULLABLE "
]
},
"execution_count": 4,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"columns().to_dataframe(search_fields=[\"description\", \"fieldName\"],search_value=\"diagnosis\")"
]
},
{
"cell_type": "markdown",
"id": "a63b4cf0",
"metadata": {},
"source": [
"\n",
"\n",
"To search the CDA, a user also needs to know what search terms are available. Each column will contain a huge amount of data, so retrieving all of the rows would be overwhelming. Instead, the CDA has a `unique_terms()` function that will return all of the unique values that populate the requested column. Like `columns`, `unique_terms` defaults to giving us an overview of the results, and can be filtered.\n",
" \n",
"
\n",
"\n",
"\n",
" \n",
"Since Julia is interested specifically in uterine cancers, she looks for columns that appear to have anatomical data, and then uses the `unique_terms` function to see what data is available for 'ResearchSubject.Diagnosis.Treatment.treatment_anatomic_site' and 'ResearchSubject.primary_diagnosis_site' to see if 'uterine' appears, she uses the `show_counts` flag so she can also see how much data is associated with each term:"
]
},
{
"cell_type": "code",
"execution_count": 5,
"id": "4527dde5",
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"\n",
" \n",
" \n",
" | treatment_anatomic_site | \n",
" Count | \n",
"
\n",
" | Loading... (need help?) |
\n",
"\n",
"\n"
],
"text/plain": [
" treatment_anatomic_site Count\n",
"0 Brain 5\n",
"1 Cervix 87\n",
"2 Head - Face Or Neck, Nos 1\n",
"3 Lymph Node(s) Paraaortic 34\n",
"4 Other 3\n",
"5 Pelvis 102\n",
"6 Spine 2\n",
"7 Unknown 125"
]
},
"execution_count": 5,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"unique_terms(\"treatment_anatomic_site\", show_counts = True).to_dataframe()"
]
},
{
"cell_type": "code",
"execution_count": 6,
"id": "740e5955",
"metadata": {
"scrolled": true
},
"outputs": [
{
"data": {
"text/html": [
"\n",
" \n",
" \n",
" | primary_diagnosis_site | \n",
" Count | \n",
"
\n",
" | Loading... (need help?) |
\n",
"\n",
"\n"
],
"text/plain": [
" primary_diagnosis_site Count\n",
"0 Abdomen 92\n",
"1 Abdomen, Mediastinum 176\n",
"2 Abdomen, Pelvis 230\n",
"3 Adrenal Glands 271\n",
"4 Adrenal gland 851\n",
".. ... ...\n",
"90 Uterus, NOS 2000\n",
"91 Vagina 72\n",
"92 Various 449\n",
"93 Various (11 locations) 89\n",
"94 Vulva 10\n",
"\n",
"[95 rows x 2 columns]"
]
},
"execution_count": 6,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"unique_terms(\"primary_diagnosis_site\", show_counts = True).to_dataframe()"
]
},
{
"cell_type": "markdown",
"id": "b005036b",
"metadata": {},
"source": [
"\n",
" \n",
"CDA makes multiple datasets searchable from a common interface, but does not harmonize the data. This means that researchers should review all the terms in a column, and not just choose the first one that fits, as there may be other similar terms available as well.\n",
" \n",
"
"
]
},
{
"cell_type": "markdown",
"id": "73e6b8dc",
"metadata": {},
"source": [
"\n",
" \n",
"Julia sees that \"treatment_anatomic_site\" does not have 'Uterine', but does have 'Cervix'. She also notes that both 'Uterus' and 'Uterus, NOS' are listed in the \"primary_diagnosis_site\" results. As she was initially looking for \"uterine\", Julia decides to expand her search a bit to account for variable naming schemes. So, she runs a fuzzy match filter on the \"ResearchSubject.primary_diagnosis_site\" for 'uter' as that should cover all variants:"
]
},
{
"cell_type": "code",
"execution_count": 7,
"id": "31064125",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"['Cervix uteri', 'Corpus uteri', 'Uterus', 'Uterus, NOS']"
]
},
"execution_count": 7,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"unique_terms(\"primary_diagnosis_site\").to_list(filters=\"uter\")"
]
},
{
"cell_type": "markdown",
"id": "9311a49e",
"metadata": {},
"source": [
"\n",
" \n",
"Just to be sure, Julia also searches for any other instances of \"cervix\":"
]
},
{
"cell_type": "code",
"execution_count": 8,
"id": "2038a8cf",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"['Cervix', 'Cervix uteri']"
]
},
"execution_count": 8,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"unique_terms(\"primary_diagnosis_site\").to_list(filters=\"cerv\")"
]
},
{
"cell_type": "markdown",
"id": "29c4de58",
"metadata": {},
"source": [
"## Building a Query\n",
"\n",
"\n",
" \n",
"With all her likely terms found, Julia begins to create a search that will get data for all of her terms. She does this by writing a series of `Q` statements that define what rows should be returned from each column. For the \"treatment_anatomic_site\", only one term is of interest, so she uses the `=` operator to get only exact matches:"
]
},
{
"cell_type": "code",
"execution_count": 9,
"id": "951fcc8f",
"metadata": {},
"outputs": [],
"source": [
"Tsite = Q('treatment_anatomic_site = \"Cervix\"')"
]
},
{
"cell_type": "markdown",
"id": "12cb5f72",
"metadata": {},
"source": [
"\n",
" \n",
"However, for \"primary_diagnosis_site\", Julia has several terms she wants to search with. Luckily, `Q` also can run fuzzy searches. It can also search more than one term at a time, so Julia writes one big `Q` statement to grab everything that is either 'uter' or 'cerv':"
]
},
{
"cell_type": "code",
"execution_count": 10,
"id": "36cfd8a4",
"metadata": {},
"outputs": [],
"source": [
"Dsite = Q('primary_diagnosis_site = \"%uter%\" OR primary_diagnosis_site = \"%cerv%\"')"
]
},
{
"cell_type": "markdown",
"id": "349af6f2",
"metadata": {},
"source": [
"\n",
" \n",
"Finally, Julia adds her two queries together into one large one:"
]
},
{
"cell_type": "code",
"execution_count": 11,
"id": "9f5f9e4f",
"metadata": {},
"outputs": [],
"source": [
"ALLDATA = Tsite.OR(Dsite)"
]
},
{
"cell_type": "markdown",
"id": "c1f5cb55",
"metadata": {},
"source": [
"## Looking at Summary Data\n",
"\n",
"\n",
" \n",
"Now that Julia has a query, she can use it to look for data in any of the CDA endpoints. She starts by getting an overall summary of what data is available using `count`:"
]
},
{
"cell_type": "code",
"execution_count": 12,
"id": "355b1706",
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"Getting results from database\n",
"\n",
"\n"
],
"text/plain": [
"Getting results from database\n",
"\n"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
"\n",
" Total execution time: 0\n",
" min 44.665 sec 44665 ms\n",
" \n",
"\n"
],
"text/plain": [
"\n",
" Total execution time: \u001b[1;36m0\u001b[0m\n",
" min \u001b[1;36m44.665\u001b[0m sec \u001b[1;36m44665\u001b[0m ms\n",
" \n"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
"specimen_count : 41069\n",
"\n"
],
"text/plain": [
"specimen_count : \u001b[1;36m41069\u001b[0m\n"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
"treatment_count : 3049\n",
"\n"
],
"text/plain": [
"treatment_count : \u001b[1;36m3049\u001b[0m\n"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
"diagnosis_count : 3685\n",
"\n"
],
"text/plain": [
"diagnosis_count : \u001b[1;36m3685\u001b[0m\n"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
"mutation_count : 903\n",
"\n"
],
"text/plain": [
"mutation_count : \u001b[1;36m903\u001b[0m\n"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
"researchsubject_count : 4869\n",
"\n"
],
"text/plain": [
"researchsubject_count : \u001b[1;36m4869\u001b[0m\n"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
"subject_count : 3742\n",
"\n"
],
"text/plain": [
"subject_count : \u001b[1;36m3742\u001b[0m\n"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": []
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/plain": []
},
"execution_count": 12,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"ALLDATA.count.run()"
]
},
{
"cell_type": "markdown",
"id": "b7ce25fc",
"metadata": {},
"source": [
"\n",
" \n",
"It seems there's a lot of data that might work for Julias study! Since she is interested in the beginings of cancer, she decides to start by looking at the researchsubject information, since that is where most of the diagnosis information is. She again gets a summary using `count`:"
]
},
{
"cell_type": "code",
"execution_count": 13,
"id": "55b0cdeb",
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"Getting results from database\n",
"\n",
"\n"
],
"text/plain": [
"Getting results from database\n",
"\n"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
"\n",
" Total execution time: 0\n",
" min 7.838 sec 7838 ms\n",
" \n",
"\n"
],
"text/plain": [
"\n",
" Total execution time: \u001b[1;36m0\u001b[0m\n",
" min \u001b[1;36m7.838\u001b[0m sec \u001b[1;36m7838\u001b[0m ms\n",
" \n"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
" files : 322958 \n",
"\n"
],
"text/plain": [
" files : \u001b[1;36m322958\u001b[0m \n"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
" total : 4869 \n",
"\n"
],
"text/plain": [
" total : \u001b[1;36m4869\u001b[0m \n"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
"\n",
"\n",
" \n",
" \n",
" | primary_diagnosis_condition | \n",
" count | \n",
"
\n",
" \n",
" \n",
" \n",
" | Adenomas and Adenocarcinomas | \n",
" 1672 | \n",
"
\n",
" \n",
" | Squamous Cell Neoplasms | \n",
" 609 | \n",
"
\n",
" \n",
" | Cystic, Mucinous and Serous Neoplasms | \n",
" 487 | \n",
"
\n",
" \n",
" | Uterine Corpus Endometrial Carcinoma | \n",
" 104 | \n",
"
\n",
" \n",
" | None | \n",
" 1175 | \n",
"
\n",
" \n",
" | Complex Mixed and Stromal Neoplasms | \n",
" 320 | \n",
"
\n",
" \n",
" | Myomatous Neoplasms | \n",
" 188 | \n",
"
\n",
" \n",
" | Epithelial Neoplasms, NOS | \n",
" 230 | \n",
"
\n",
" \n",
" | Not Reported | \n",
" 12 | \n",
"
\n",
" \n",
" | Complex Epithelial Neoplasms | \n",
" 27 | \n",
"
\n",
" \n",
" | Neuroepitheliomatous Neoplasms | \n",
" 1 | \n",
"
\n",
" \n",
" | Soft Tissue Tumors and Sarcomas, NOS | \n",
" 14 | \n",
"
\n",
" \n",
" | Mesonephromas | \n",
" 5 | \n",
"
\n",
" \n",
" | Neoplasms, NOS | \n",
" 12 | \n",
"
\n",
" \n",
" | Trophoblastic neoplasms | \n",
" 13 | \n",
"
\n",
" \n",
"
\n",
"\n",
"\n",
" \n",
" \n",
" | primary_diagnosis_site | \n",
" count | \n",
"
\n",
" \n",
" \n",
" \n",
" | Uterus, NOS | \n",
" 2000 | \n",
"
\n",
" \n",
" | Corpus uteri | \n",
" 780 | \n",
"
\n",
" \n",
" | Cervix uteri | \n",
" 915 | \n",
"
\n",
" \n",
" | Uterus | \n",
" 867 | \n",
"
\n",
" \n",
" | Cervix | \n",
" 307 | \n",
"
\n",
" \n",
"
\n",
"\n",
"\n",
" \n",
" \n",
" | researchsubject_identifier_system | \n",
" count | \n",
"
\n",
" \n",
" \n",
" \n",
" | GDC | \n",
" 3591 | \n",
"
\n",
" \n",
" | PDC | \n",
" 104 | \n",
"
\n",
" \n",
" | IDC | \n",
" 1174 | \n",
"
\n",
" \n",
"
\n"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/plain": []
},
"execution_count": 13,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"ALLDATA.researchsubject.count.run()"
]
},
{
"cell_type": "markdown",
"id": "86a323e2",
"metadata": {},
"source": [
"## Refining Queries\n",
"\n",
"\n",
" \n",
"Browsing the primary_diagnosis_condition data, Julia notices that there are a large number of research subjects that are Adenomas and Adenocarcinomas. Since Julia wants to look for common phenotypes in early cancers, she decides it might be easier to exclude the endocrine related data, as they might have different mechanisms. So she adds a new filter to her query:"
]
},
{
"cell_type": "code",
"execution_count": 14,
"id": "0d526198",
"metadata": {
"scrolled": true
},
"outputs": [
{
"data": {
"text/html": [
"Getting results from database\n",
"\n",
"\n"
],
"text/plain": [
"Getting results from database\n",
"\n"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
"\n",
" Total execution time: 0\n",
" min 7.824 sec 7824 ms\n",
" \n",
"\n"
],
"text/plain": [
"\n",
" Total execution time: \u001b[1;36m0\u001b[0m\n",
" min \u001b[1;36m7.824\u001b[0m sec \u001b[1;36m7824\u001b[0m ms\n",
" \n"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
" files : 297379 \n",
"\n"
],
"text/plain": [
" files : \u001b[1;36m297379\u001b[0m \n"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
" total : 3197 \n",
"\n"
],
"text/plain": [
" total : \u001b[1;36m3197\u001b[0m \n"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
"\n",
"\n",
" \n",
" \n",
" | primary_diagnosis_condition | \n",
" count | \n",
"
\n",
" \n",
" \n",
" \n",
" | Myomatous Neoplasms | \n",
" 188 | \n",
"
\n",
" \n",
" | Uterine Corpus Endometrial Carcinoma | \n",
" 104 | \n",
"
\n",
" \n",
" | Cystic, Mucinous and Serous Neoplasms | \n",
" 487 | \n",
"
\n",
" \n",
" | Squamous Cell Neoplasms | \n",
" 609 | \n",
"
\n",
" \n",
" | Complex Mixed and Stromal Neoplasms | \n",
" 320 | \n",
"
\n",
" \n",
" | None | \n",
" 1175 | \n",
"
\n",
" \n",
" | Epithelial Neoplasms, NOS | \n",
" 230 | \n",
"
\n",
" \n",
" | Soft Tissue Tumors and Sarcomas, NOS | \n",
" 14 | \n",
"
\n",
" \n",
" | Mesonephromas | \n",
" 5 | \n",
"
\n",
" \n",
" | Neoplasms, NOS | \n",
" 12 | \n",
"
\n",
" \n",
" | Trophoblastic neoplasms | \n",
" 13 | \n",
"
\n",
" \n",
" | Complex Epithelial Neoplasms | \n",
" 27 | \n",
"
\n",
" \n",
" | Not Reported | \n",
" 12 | \n",
"
\n",
" \n",
" | Neuroepitheliomatous Neoplasms | \n",
" 1 | \n",
"
\n",
" \n",
"
\n",
"\n",
"\n",
" \n",
" \n",
" | primary_diagnosis_site | \n",
" count | \n",
"
\n",
" \n",
" \n",
" \n",
" | Corpus uteri | \n",
" 373 | \n",
"
\n",
" \n",
" | Uterus, NOS | \n",
" 962 | \n",
"
\n",
" \n",
" | Cervix uteri | \n",
" 688 | \n",
"
\n",
" \n",
" | Uterus | \n",
" 867 | \n",
"
\n",
" \n",
" | Cervix | \n",
" 307 | \n",
"
\n",
" \n",
"
\n",
"\n",
"\n",
" \n",
" \n",
" | researchsubject_identifier_system | \n",
" count | \n",
"
\n",
" \n",
" \n",
" \n",
" | GDC | \n",
" 1919 | \n",
"
\n",
" \n",
" | PDC | \n",
" 104 | \n",
"
\n",
" \n",
" | IDC | \n",
" 1174 | \n",
"
\n",
" \n",
"
\n"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/plain": []
},
"execution_count": 14,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"Noadeno = Q('primary_diagnosis_condition != \"Adenomas and Adenocarcinomas\"')\n",
"\n",
"NoAdenoData = ALLDATA.AND(Noadeno)\n",
"\n",
"NoAdenoData.researchsubject.count.run()"
]
},
{
"cell_type": "markdown",
"id": "40a0191d",
"metadata": {},
"source": [
"\n",
" \n",
"She then previews the actual metadata for researchsubject, subject, and file, to make sure that they have all the information she will need for her work. Since she's mostly interested in looking at the kinds of data available from each endpoint:"
]
},
{
"cell_type": "code",
"execution_count": 15,
"id": "d186b837",
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"Getting results from database\n",
"\n",
"\n"
],
"text/plain": [
"Getting results from database\n",
"\n"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
"\n",
" Total execution time: 0\n",
" min 4.19 sec 4190 ms\n",
" \n",
"\n"
],
"text/plain": [
"\n",
" Total execution time: \u001b[1;36m0\u001b[0m\n",
" min \u001b[1;36m4.19\u001b[0m sec \u001b[1;36m4190\u001b[0m ms\n",
" \n"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
"\n",
" \n",
" \n",
" | researchsubject_id | \n",
" researchsubject_identifier | \n",
" member_of_research_project | \n",
" primary_diagnosis_condition | \n",
" primary_diagnosis_site | \n",
" subject_id | \n",
"
\n",
" | Loading... (need help?) |
\n",
"\n",
"\n"
],
"text/plain": [
" researchsubject_id \\\n",
"0 03b6910a-f8f1-43cf-afe8-8b9ea2687b26 \n",
"1 2317cb92-3406-4974-a2ce-04307b17fc52 \n",
"2 2e08a89e-4035-4782-b54f-f98eef5db80d \n",
"3 31187375-5a0a-4e60-837b-cd5395a6971e \n",
"4 3223fec6-1d35-419a-8ee9-7c5885b8be1e \n",
".. ... \n",
"95 TCGA-VS-A9UH__tcga_cesc \n",
"96 a307e88b-745c-469e-ba24-a7cc544e8859 \n",
"97 d4510801-8ce4-4f10-8b31-25c0a39b5135 \n",
"98 d9952e44-b573-4854-893e-b3bbc5287f2f \n",
"99 de4e13c8-5ad0-46d2-ae54-b9ca22b939d8 \n",
"\n",
" researchsubject_identifier \\\n",
"0 [{'system': 'GDC', 'value': '03b6910a-f8f1-43c... \n",
"1 [{'system': 'GDC', 'value': '2317cb92-3406-497... \n",
"2 [{'system': 'GDC', 'value': '2e08a89e-4035-478... \n",
"3 [{'system': 'GDC', 'value': '31187375-5a0a-4e6... \n",
"4 [{'system': 'GDC', 'value': '3223fec6-1d35-419... \n",
".. ... \n",
"95 [{'system': 'IDC', 'value': 'TCGA-VS-A9UH__tcg... \n",
"96 [{'system': 'GDC', 'value': 'a307e88b-745c-469... \n",
"97 [{'system': 'GDC', 'value': 'd4510801-8ce4-4f1... \n",
"98 [{'system': 'GDC', 'value': 'd9952e44-b573-485... \n",
"99 [{'system': 'GDC', 'value': 'de4e13c8-5ad0-46d... \n",
"\n",
" member_of_research_project primary_diagnosis_condition \\\n",
"0 GENIE-MSK Epithelial Neoplasms, NOS \n",
"1 GENIE-MSK Complex Mixed and Stromal Neoplasms \n",
"2 TCGA-CESC Squamous Cell Neoplasms \n",
"3 GENIE-MSK Complex Epithelial Neoplasms \n",
"4 TCGA-CESC Squamous Cell Neoplasms \n",
".. ... ... \n",
"95 tcga_cesc None \n",
"96 GENIE-DFCI Complex Mixed and Stromal Neoplasms \n",
"97 TCGA-CESC Squamous Cell Neoplasms \n",
"98 TCGA-UCS Complex Mixed and Stromal Neoplasms \n",
"99 TCGA-UCEC Cystic, Mucinous and Serous Neoplasms \n",
"\n",
" primary_diagnosis_site subject_id \n",
"0 Corpus uteri GENIE-MSK-P-0018443 \n",
"1 Uterus, NOS GENIE-MSK-P-0018286 \n",
"2 Cervix uteri TCGA-VS-A8EJ \n",
"3 Cervix uteri GENIE-MSK-P-0001655 \n",
"4 Cervix uteri TCGA-LP-A4AX \n",
".. ... ... \n",
"95 Cervix TCGA-VS-A9UH \n",
"96 Corpus uteri GENIE-DFCI-023832 \n",
"97 Cervix uteri TCGA-EK-A2R8 \n",
"98 Uterus, NOS TCGA-N5-A4RD \n",
"99 Corpus uteri TCGA-EO-A1Y8 \n",
"\n",
"[100 rows x 6 columns]"
]
},
"execution_count": 15,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"NoAdenoData.researchsubject.run().to_dataframe() # view the dataframe"
]
},
{
"cell_type": "markdown",
"id": "086697b3",
"metadata": {},
"source": [
"---\n",
"\n",
"\n",
"\n",
"
ResearchSubject Field Definitions
\n",
"\n",
"
A research subject is the entity of interest in a specific research study or project, typically a human being or an animal, but can also be a device, group of humans or animals, or a tissue sample. Human research subjects are usually not traceable to a particular person to protect the subjects privacy. This entity plays the role of the case_id in existing data. An individual who participates in 3 studies will have 3 researchsubject IDs\n",
" \n",
"
\n",
"- id: The 'logical' identifier of the entity in the system of record, e.g. a UUID. This 'id' is unique within a given system. The identified entity may have a different 'id' in a different system. For CDA, this is case_id.
\n",
"- identifier: A 'business' identifier for the entity, typically as provided by an external system or authority, that persists across implementing systems (i.e. a 'logical' identifier). Uses a specialized, complex 'Identifier' data type to capture information about the source of the business identifier - or a URI expressed as a string to an existing entity.
\n",
"- identifier.system: The system or namespace that defines the identifier.
\n",
"- identifier.value: The value of the identifier, as defined by the system.
\n",
"- member_of_research_project: A reference to the Study(s) of which this ResearchSubject is a member.
\n",
"- primary_diagnosis_condition: The text term used to describe the type of malignant disease, as categorized by the World Health Organization's (WHO) International Classification of Diseases for Oncology (ICD-O). This attribute represents the disease that qualified the subject for inclusion on the ResearchProject.
\n",
"- primary_diagnosis_site: The text term used to describe the primary site of disease, as categorized by the World Health Organization's (WHO) International Classification of Diseases for Oncology (ICD-O). This categorization groups cases into general categories. This attribute represents the primary site of disease that qualified the subject for inclusion on the ResearchProject.
\n",
"- subject_id: The 'logical' identifier of the entity in the system of record, e.g. a UUID. This 'id' is unique within a given system. The identified entity may have a different 'id' in a different system. Can be joined to the `id` field from subject results
\n",
"
\n",
"\n",
"
Getting results from database\n",
"\n",
"\n"
],
"text/plain": [
"Getting results from database\n",
"\n"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
"\n",
" Total execution time: 0\n",
" min 4.263 sec 4263 ms\n",
" \n",
"\n"
],
"text/plain": [
"\n",
" Total execution time: \u001b[1;36m0\u001b[0m\n",
" min \u001b[1;36m4.263\u001b[0m sec \u001b[1;36m4263\u001b[0m ms\n",
" \n"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
"\n",
" \n",
" \n",
" | subject_id | \n",
" subject_identifier | \n",
" species | \n",
" sex | \n",
" race | \n",
" ethnicity | \n",
" days_to_birth | \n",
" subject_associated_project | \n",
" vital_status | \n",
" days_to_death | \n",
" cause_of_death | \n",
"
\n",
" | Loading... (need help?) |
\n",
"\n",
"\n"
],
"text/plain": [
" subject_id subject_identifier \\\n",
"0 AD10485 [{'system': 'GDC', 'value': 'AD10485'}] \n",
"1 AD17572 [{'system': 'GDC', 'value': 'AD17572'}] \n",
"2 AD6453 [{'system': 'GDC', 'value': 'AD6453'}] \n",
"3 AD6896 [{'system': 'GDC', 'value': 'AD6896'}] \n",
"4 C3L-00586 [{'system': 'GDC', 'value': 'C3L-00586'}, {'sy... \n",
".. ... ... \n",
"95 TCGA-EY-A1GP [{'system': 'GDC', 'value': 'TCGA-EY-A1GP'}, {... \n",
"96 TCGA-FI-A2D2 [{'system': 'GDC', 'value': 'TCGA-FI-A2D2'}, {... \n",
"97 TCGA-N9-A4PZ [{'system': 'GDC', 'value': 'TCGA-N9-A4PZ'}, {... \n",
"98 TCGA-PG-A6IB [{'system': 'GDC', 'value': 'TCGA-PG-A6IB'}, {... \n",
"99 TCGA-VS-A8QC [{'system': 'GDC', 'value': 'TCGA-VS-A8QC'}, {... \n",
"\n",
" species sex race ethnicity \\\n",
"0 homo sapiens female not reported not reported \n",
"1 homo sapiens female not reported not reported \n",
"2 homo sapiens female not reported not reported \n",
"3 homo sapiens female not reported not reported \n",
"4 homo sapiens female white not hispanic or latino \n",
".. ... ... ... ... \n",
"95 homo sapiens female black or african american not hispanic or latino \n",
"96 homo sapiens female white not hispanic or latino \n",
"97 homo sapiens female white not reported \n",
"98 homo sapiens female black or african american not hispanic or latino \n",
"99 homo sapiens female white not reported \n",
"\n",
" days_to_birth subject_associated_project vital_status \\\n",
"0 NaN [FM-AD] Not Reported \n",
"1 NaN [FM-AD] Not Reported \n",
"2 NaN [FM-AD] Not Reported \n",
"3 NaN [FM-AD] Not Reported \n",
"4 -18580.0 [CPTAC-3, cptac_ucec, CPTAC3-Discovery] Alive \n",
".. ... ... ... \n",
"95 NaN [TCGA-UCEC, tcga_ucec] Alive \n",
"96 NaN [TCGA-UCEC, tcga_ucec] Dead \n",
"97 NaN [TCGA-UCS, tcga_ucs] Dead \n",
"98 NaN [TCGA-UCEC, tcga_ucec] Alive \n",
"99 NaN [TCGA-CESC, tcga_cesc] Dead \n",
"\n",
" days_to_death cause_of_death \n",
"0 None None \n",
"1 None None \n",
"2 None None \n",
"3 None None \n",
"4 None Not Reported \n",
".. ... ... \n",
"95 None None \n",
"96 None None \n",
"97 None None \n",
"98 None None \n",
"99 None None \n",
"\n",
"[100 rows x 11 columns]"
]
},
"execution_count": 16,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"NoAdenoData.subject.run().to_dataframe() # view the dataframe"
]
},
{
"cell_type": "markdown",
"id": "dec76132",
"metadata": {},
"source": [
"---\n",
"\n",
"\n",
"\n",
"
Subject Field Definitions
\n",
"\n",
"
A patient entity captures the study-independent metadata for research subjects. Human research subjects are usually not traceable to a particular person to protect the subjects privacy.\n",
"\n",
" \n",
"
\n",
"- id: The 'logical' identifier of the entity in the system of record, e.g. a UUID. This 'id' is unique within a given system. The identified entity may have a different 'id' in a different system.\",STRING
\n",
"- identifier: A 'business' identifier for the entity, typically as provided by an external system or authority, that persists across implementing systems (i.e. a 'logical' identifier). Uses a specialized, complex 'Identifier' data type to capture information about the source of the business identifier - or a URI expressed as a string to an existing entity.
\n",
"- identifier.system: The system or namespace that defines the identifier.
\n",
"- identifier.value: The value of the identifier, as defined by the system.
\n",
"- species: The taxonomic group (e.g. species) of the patient. For MVP, since taxonomy vocabulary is consistent between GDC and PDC, using text. Ultimately, this will be a term returned by the vocabulary service.
\n",
"- sex: The biologic character or quality that distinguishes male and female from one another as expressed by analysis of the person's gonadal, morphologic (internal and external), chromosomal, and hormonal characteristics.
\n",
"- race: An arbitrary classification of a taxonomic group that is a division of a species. It usually arises as a consequence of geographical isolation within a species and is characterized by shared heredity, physical attributes and behavior, and in the case of humans, by common history, nationality, or geographic distribution. The provided values are based on the categories defined by the U.S. Office of Management and Business and used by the U.S. Census Bureau.
\n",
"- ethnicity: An individual's self-described social and cultural grouping, specifically whether an individual describes themselves as Hispanic or Latino. The provided values are based on the categories defined by the U.S. Office of Management and Business and used by the U.S. Census Bureau.
\n",
"- days_to_birth: Number of days between the date used for index and the date from a person's date of birth represented as a calculated negative number of days.
\n",
"- subject_associated_project: The list of Projects associated with the Subject.
\n",
"- vital_status: Coded value indicating the state or condition of being living or deceased; also includes the case where the vital status is unknown.
\n",
"- days_to_death: Number of days between the date used for index and the date from a person's date of death represented as a calculated number of days.
\n",
"- cause_of_death: Coded value indicating the circumstance or condition that results in the death of the subject.
\n",
"
\n",
"\n",
"
Getting results from database\n",
"\n",
"\n"
],
"text/plain": [
"Getting results from database\n",
"\n"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
"\n",
" Total execution time: 0\n",
" min 13.387 sec 13387 ms\n",
" \n",
"\n"
],
"text/plain": [
"\n",
" Total execution time: \u001b[1;36m0\u001b[0m\n",
" min \u001b[1;36m13.387\u001b[0m sec \u001b[1;36m13387\u001b[0m ms\n",
" \n"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
"\n",
" \n",
" \n",
" | file_id | \n",
" file_identifier | \n",
" label | \n",
" data_category | \n",
" data_type | \n",
" file_format | \n",
" file_associated_project | \n",
" drs_uri | \n",
" byte_size | \n",
" checksum | \n",
" data_modality | \n",
" imaging_modality | \n",
" dbgap_accession_number | \n",
" imaging_series | \n",
" subject_id | \n",
"
\n",
" | Loading... (need help?) |
\n",
"\n",
"\n"
],
"text/plain": [
" file_id \\\n",
"0 067a1e40-3b7f-4058-863e-c3f246e80029 \n",
"1 073249ac-a3ea-46ca-a89e-82e26e3582ce \n",
"2 192273a6-3485-450b-a596-33ee9c67cd59 \n",
"3 1b99980e-9ea2-4101-94eb-2edcff38b2e3 \n",
"4 28649881-c6cb-419b-a5cf-ec596cc007c0 \n",
".. ... \n",
"95 2cf0dee5-0306-4b7e-b895-4cdcdd5d3520 \n",
"96 3bf04ede-73ab-4637-9bec-17303d6c9072 \n",
"97 467d743f-af87-4248-9de0-732821f03b0f \n",
"98 54f5b813-cfe4-4aae-9124-99f82043f546 \n",
"99 57b94fd1-a8a0-4879-8647-1947c742ea4c \n",
"\n",
" file_identifier \\\n",
"0 [{'system': 'GDC', 'value': '067a1e40-3b7f-405... \n",
"1 [{'system': 'IDC', 'value': '073249ac-a3ea-46c... \n",
"2 [{'system': 'GDC', 'value': '192273a6-3485-450... \n",
"3 [{'system': 'IDC', 'value': '1b99980e-9ea2-410... \n",
"4 [{'system': 'GDC', 'value': '28649881-c6cb-419... \n",
".. ... \n",
"95 [{'system': 'GDC', 'value': '2cf0dee5-0306-4b7... \n",
"96 [{'system': 'IDC', 'value': '3bf04ede-73ab-463... \n",
"97 [{'system': 'GDC', 'value': '467d743f-af87-424... \n",
"98 [{'system': 'GDC', 'value': '54f5b813-cfe4-4aa... \n",
"99 [{'system': 'GDC', 'value': '57b94fd1-a8a0-487... \n",
"\n",
" label \\\n",
"0 091bd25a-a6f0-4b2b-bd52-d08405b662ca.genie.ali... \n",
"1 073249ac-a3ea-46ca-a89e-82e26e3582ce.dcm \n",
"2 55c118ac-28ef-4c05-b0c2-42520fcb62cd.genie.ali... \n",
"3 1b99980e-9ea2-4101-94eb-2edcff38b2e3.dcm \n",
"4 e1a62968-df48-425e-831f-59470d37ce9f.targeted_... \n",
".. ... \n",
"95 f658ff92-8cc4-4f07-9cbf-7f9c9544db73.targeted_... \n",
"96 3bf04ede-73ab-4637-9bec-17303d6c9072.dcm \n",
"97 b03017a8-dd14-48cd-96e2-70d1a623f240.genie.ali... \n",
"98 4418d94c-a2c6-4066-87ef-4417c6bb44be.genie.ali... \n",
"99 080496d4-9f5b-4c94-887f-6ab9fece68d9.targeted_... \n",
"\n",
" data_category data_type \\\n",
"0 Simple Nucleotide Variation Masked Annotated Somatic Mutation \n",
"1 Imaging None \n",
"2 Copy Number Variation Gene Level Copy Number Scores \n",
"3 Imaging None \n",
"4 Simple Nucleotide Variation Raw Simple Somatic Mutation \n",
".. ... ... \n",
"95 Simple Nucleotide Variation Aggregated Somatic Mutation \n",
"96 Imaging None \n",
"97 Copy Number Variation Gene Level Copy Number Scores \n",
"98 Simple Nucleotide Variation Masked Annotated Somatic Mutation \n",
"99 Simple Nucleotide Variation Aggregated Somatic Mutation \n",
"\n",
" file_format file_associated_project \\\n",
"0 MAF [GENIE-MSK] \n",
"1 DICOM [tcga_ucs] \n",
"2 TSV [GENIE-DFCI] \n",
"3 DICOM [tcga_ucs] \n",
"4 VCF [FM-AD] \n",
".. ... ... \n",
"95 MAF [FM-AD] \n",
"96 DICOM [tcga_ucs] \n",
"97 TSV [GENIE-DFCI] \n",
"98 MAF [GENIE-MSK] \n",
"99 MAF [FM-AD] \n",
"\n",
" drs_uri byte_size \\\n",
"0 drs://dg.4DFC:067a1e40-3b7f-4058-863e-c3f246e8... 1707.0 \n",
"1 drs://dg.4DFC:073249ac-a3ea-46ca-a89e-82e26e35... NaN \n",
"2 drs://dg.4DFC:192273a6-3485-450b-a596-33ee9c67... 22910.0 \n",
"3 drs://dg.4DFC:1b99980e-9ea2-4101-94eb-2edcff38... NaN \n",
"4 drs://dg.4DFC:28649881-c6cb-419b-a5cf-ec596cc0... 1142.0 \n",
".. ... ... \n",
"95 drs://dg.4DFC:2cf0dee5-0306-4b7e-b895-4cdcdd5d... 2989.0 \n",
"96 drs://dg.4DFC:3bf04ede-73ab-4637-9bec-17303d6c... NaN \n",
"97 drs://dg.4DFC:467d743f-af87-4248-9de0-732821f0... 22910.0 \n",
"98 drs://dg.4DFC:54f5b813-cfe4-4aae-9124-99f82043... 4228.0 \n",
"99 drs://dg.4DFC:57b94fd1-a8a0-4879-8647-1947c742... 5231.0 \n",
"\n",
" checksum data_modality imaging_modality \\\n",
"0 d749b97ca11259fb3d901ac8b5aa6b87 Genomic None \n",
"1 None Imaging SM \n",
"2 382eb1d6089aa55be3ddda8241bc1a8b Genomic None \n",
"3 None Imaging SM \n",
"4 96465673e159f84aab29db86b6955d56 Genomic None \n",
".. ... ... ... \n",
"95 47ee03122ba0bbd1a15b2b6d86dfde2a Genomic None \n",
"96 None Imaging SM \n",
"97 58b2459c6a8010dee5bf7af37fd53d6f Genomic None \n",
"98 325cb5e9e8e2e3cf7a9ce95f86cecda4 Genomic None \n",
"99 29a544e56079b74aed6f665cb0f79945 Genomic None \n",
"\n",
" dbgap_accession_number imaging_series \\\n",
"0 None None \n",
"1 None 0e9df1a6-ab24-4937-9a08-2d807b1b3f60 \n",
"2 None None \n",
"3 None 7cc334ae-978b-44f0-872e-70a28c1274a0 \n",
"4 phs001179 None \n",
".. ... ... \n",
"95 phs001179 None \n",
"96 None a7201e5d-94a4-4df2-901e-a38963025f4f \n",
"97 None None \n",
"98 None None \n",
"99 phs001179 None \n",
"\n",
" subject_id \n",
"0 GENIE-MSK-P-0016598 \n",
"1 TCGA-N8-A4PM \n",
"2 GENIE-DFCI-020078 \n",
"3 TCGA-N7-A59B \n",
"4 AD15035 \n",
".. ... \n",
"95 AD8297 \n",
"96 TCGA-ND-A4WC \n",
"97 GENIE-DFCI-005105 \n",
"98 GENIE-MSK-P-0017720 \n",
"99 AD5188 \n",
"\n",
"[100 rows x 15 columns]"
]
},
"execution_count": 17,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"NoAdenoData.subject.file.run().to_dataframe() # view the dataframe"
]
},
{
"cell_type": "markdown",
"id": "8cf9f2d3",
"metadata": {},
"source": [
"\n",
"---\n",
"\n",
"\n",
"\n",
"
File Field Definitions
\n",
"\n",
"
A file is an information-bearing electronic object that contains a physical embodiment of some information using a particular character encoding.\n",
"\n",
" \n",
"
\n",
" - id: The 'logical' identifier of the entity in the system of record, e.g. a UUID. This 'id' is unique within a given system. The identified entity may have a different 'id' in a different system.
\n",
"- identifier: A 'business' identifier for the entity, typically as provided by an external system or authority, that persists across implementing systems (i.e. a 'logical' identifier). Uses a specialized, complex 'Identifier' data type to capture information about the source of the business identifier - or a URI expressed as a string to an existing entity.
\n",
"- identifier.system: The system or namespace that defines the identifier.
\n",
"- identifier.value: The value of the identifier, as defined by the system.
\n",
" - label: Short name or abbreviation for dataset. Maps to rdfs:label.
\n",
" - data_catagory: Broad categorization of the contents of the data file.
\n",
" - data_type: Specific content type of the data file.
\n",
" - file_format: Format of the data files.
\n",
" - associated_project: A reference to the Project(s) of which this ResearchSubject is a member. The associated_project may be embedded using the ref definition or may be a reference to the id for the Project - or a URI expressed as a string to an existing entity.
\n",
" - drs_uri: A string of characters used to identify a resource on the Data Repo Service(DRS). Can be used to retrieve this specific file from a server.
\n",
" - byte_size: Size of the file in bytes. Maps to dcat:byteSize.
\n",
" - checksum: The md5 value for the file. A digit representing the sum of the correct digits in a piece of stored or transmitted digital data, against which later comparisons can be made to detect errors in the data.
\n",
" - data_modality: Data modality describes the biological nature of the information gathered as the result of an Activity, independent of the technology or methods used to produce the information. Always one of \"Genomic\", \"Proteomic\", or \"Imaging\".
\n",
" - imaging_modality: An imaging modality describes the imaging equipment and/or method used to acquire certain structural or functional information about the body. These include but are not limited to computed tomography (CT) and magnetic resonance imaging (MRI). Taken from the DICOM standard.
\n",
" - dbgap_accession_number: The dbgap accession number for the project.
\n",
"
\n",
"\n",
"
Getting results from database\n",
"\n",
"\n"
],
"text/plain": [
"Getting results from database\n",
"\n"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
"\n",
" Total execution time: 0\n",
" min 3.703 sec 3703 ms\n",
" \n",
"\n"
],
"text/plain": [
"\n",
" Total execution time: \u001b[1;36m0\u001b[0m\n",
" min \u001b[1;36m3.703\u001b[0m sec \u001b[1;36m3703\u001b[0m ms\n",
" \n"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "310075429a814eb78eb14da40f6587aa",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"Output()"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
"\n"
],
"text/plain": []
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
"\n",
"\n"
],
"text/plain": [
"\n"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
"\n",
"\n"
],
"text/plain": [
"\n"
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"researchsubs = NoAdenoData.researchsubject.run()\n",
"rsdf = researchsubs.auto_paginator(to_df=True)"
]
},
{
"cell_type": "code",
"execution_count": 19,
"id": "a1258057",
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"Getting results from database\n",
"\n",
"\n"
],
"text/plain": [
"Getting results from database\n",
"\n"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
"\n",
" Total execution time: 0\n",
" min 3.642 sec 3642 ms\n",
" \n",
"\n"
],
"text/plain": [
"\n",
" Total execution time: \u001b[1;36m0\u001b[0m\n",
" min \u001b[1;36m3.642\u001b[0m sec \u001b[1;36m3642\u001b[0m ms\n",
" \n"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "5fdf1238d24a4f2185842418da83e976",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"Output()"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
"\n"
],
"text/plain": []
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
"\n",
"\n"
],
"text/plain": [
"\n"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
"\n",
"\n"
],
"text/plain": [
"\n"
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"subs = NoAdenoData.subject.run()\n",
"subsdf = subs.auto_paginator(to_df=True)"
]
},
{
"cell_type": "code",
"execution_count": 20,
"id": "04cd73df",
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"\n",
" \n",
" | \n",
" researchsubject_id | \n",
" researchsubject_identifier | \n",
" member_of_research_project | \n",
" primary_diagnosis_condition | \n",
" primary_diagnosis_site | \n",
" subject_id | \n",
"
\n",
" | Loading... (need help?) |
\n",
"\n",
"\n"
],
"text/plain": [
" researchsubject_id \\\n",
"0 d6929fe7-908d-4684-b799-1c7d1bfe4835 \n",
"1 e9de2706-246c-46ae-a48c-7cb9bff7e723 \n",
"2 0cabe364-1e30-433d-b301-3fcffd953b09 \n",
"3 14c744c0-1ae8-4d8b-bb7c-6149019dc1a3 \n",
"4 164241df-c75b-482d-ad6a-84b0ee9aaf66 \n",
".. ... \n",
"95 TCGA-VS-A9UH__tcga_cesc \n",
"96 a307e88b-745c-469e-ba24-a7cc544e8859 \n",
"97 d4510801-8ce4-4f10-8b31-25c0a39b5135 \n",
"98 d9952e44-b573-4854-893e-b3bbc5287f2f \n",
"99 de4e13c8-5ad0-46d2-ae54-b9ca22b939d8 \n",
"\n",
" researchsubject_identifier \\\n",
"0 [{'system': 'GDC', 'value': 'd6929fe7-908d-468... \n",
"1 [{'system': 'GDC', 'value': 'e9de2706-246c-46a... \n",
"2 [{'system': 'GDC', 'value': '0cabe364-1e30-433... \n",
"3 [{'system': 'GDC', 'value': '14c744c0-1ae8-4d8... \n",
"4 [{'system': 'GDC', 'value': '164241df-c75b-482... \n",
".. ... \n",
"95 [{'system': 'IDC', 'value': 'TCGA-VS-A9UH__tcg... \n",
"96 [{'system': 'GDC', 'value': 'a307e88b-745c-469... \n",
"97 [{'system': 'GDC', 'value': 'd4510801-8ce4-4f1... \n",
"98 [{'system': 'GDC', 'value': 'd9952e44-b573-485... \n",
"99 [{'system': 'GDC', 'value': 'de4e13c8-5ad0-46d... \n",
"\n",
" member_of_research_project primary_diagnosis_condition \\\n",
"0 GENIE-MSK Myomatous Neoplasms \n",
"1 GENIE-DFCI Epithelial Neoplasms, NOS \n",
"2 FM-AD Cystic, Mucinous and Serous Neoplasms \n",
"3 GENIE-UHN Epithelial Neoplasms, NOS \n",
"4 GENIE-DFCI Complex Mixed and Stromal Neoplasms \n",
".. ... ... \n",
"95 tcga_cesc None \n",
"96 GENIE-DFCI Complex Mixed and Stromal Neoplasms \n",
"97 TCGA-CESC Squamous Cell Neoplasms \n",
"98 TCGA-UCS Complex Mixed and Stromal Neoplasms \n",
"99 TCGA-UCEC Cystic, Mucinous and Serous Neoplasms \n",
"\n",
" primary_diagnosis_site subject_id \n",
"0 Uterus, NOS GENIE-MSK-P-0003754 \n",
"1 Corpus uteri GENIE-DFCI-036994 \n",
"2 Uterus, NOS AD10783 \n",
"3 Corpus uteri GENIE-UHN-944776 \n",
"4 Corpus uteri GENIE-DFCI-009119 \n",
".. ... ... \n",
"95 Cervix TCGA-VS-A9UH \n",
"96 Corpus uteri GENIE-DFCI-023832 \n",
"97 Cervix uteri TCGA-EK-A2R8 \n",
"98 Uterus, NOS TCGA-N5-A4RD \n",
"99 Corpus uteri TCGA-EO-A1Y8 \n",
"\n",
"[3197 rows x 6 columns]"
]
},
"execution_count": 20,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"rsdf # view the researchsubject dataframe"
]
},
{
"cell_type": "code",
"execution_count": 21,
"id": "92a6f811",
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"\n",
" \n",
" | \n",
" subject_id | \n",
" subject_identifier | \n",
" species | \n",
" sex | \n",
" race | \n",
" ethnicity | \n",
" days_to_birth | \n",
" subject_associated_project | \n",
" vital_status | \n",
" days_to_death | \n",
" cause_of_death | \n",
"
\n",
" | Loading... (need help?) |
\n",
"\n",
"\n"
],
"text/plain": [
" subject_id subject_identifier \\\n",
"0 HTMCP-03-06-02146 [{'system': 'GDC', 'value': 'HTMCP-03-06-02146'}] \n",
"1 TCGA-AX-A0IU [{'system': 'GDC', 'value': 'TCGA-AX-A0IU'}, {... \n",
"2 TCGA-AX-A1C5 [{'system': 'GDC', 'value': 'TCGA-AX-A1C5'}, {... \n",
"3 TCGA-AX-A1CA [{'system': 'GDC', 'value': 'TCGA-AX-A1CA'}, {... \n",
"4 TCGA-AX-A1CF [{'system': 'GDC', 'value': 'TCGA-AX-A1CF'}, {... \n",
".. ... ... \n",
"95 TCGA-EY-A1GP [{'system': 'GDC', 'value': 'TCGA-EY-A1GP'}, {... \n",
"96 TCGA-FI-A2D2 [{'system': 'GDC', 'value': 'TCGA-FI-A2D2'}, {... \n",
"97 TCGA-N9-A4PZ [{'system': 'GDC', 'value': 'TCGA-N9-A4PZ'}, {... \n",
"98 TCGA-PG-A6IB [{'system': 'GDC', 'value': 'TCGA-PG-A6IB'}, {... \n",
"99 TCGA-VS-A8QC [{'system': 'GDC', 'value': 'TCGA-VS-A8QC'}, {... \n",
"\n",
" species sex race ethnicity \\\n",
"0 homo sapiens female black or african american not hispanic or latino \n",
"1 homo sapiens female black or african american not hispanic or latino \n",
"2 homo sapiens female white not hispanic or latino \n",
"3 homo sapiens female black or african american not hispanic or latino \n",
"4 homo sapiens female white not hispanic or latino \n",
".. ... ... ... ... \n",
"95 homo sapiens female black or african american not hispanic or latino \n",
"96 homo sapiens female white not hispanic or latino \n",
"97 homo sapiens female white not reported \n",
"98 homo sapiens female black or african american not hispanic or latino \n",
"99 homo sapiens female white not reported \n",
"\n",
" days_to_birth subject_associated_project vital_status days_to_death \\\n",
"0 NaN [CGCI-HTMCP-CC] Dead None \n",
"1 NaN [TCGA-UCEC, tcga_ucec] Alive None \n",
"2 NaN [TCGA-UCEC, tcga_ucec] Alive None \n",
"3 NaN [TCGA-UCEC, tcga_ucec] Alive None \n",
"4 NaN [TCGA-UCEC, tcga_ucec] Alive None \n",
".. ... ... ... ... \n",
"95 NaN [TCGA-UCEC, tcga_ucec] Alive NaN \n",
"96 NaN [TCGA-UCEC, tcga_ucec] Dead NaN \n",
"97 NaN [TCGA-UCS, tcga_ucs] Dead NaN \n",
"98 NaN [TCGA-UCEC, tcga_ucec] Alive NaN \n",
"99 NaN [TCGA-CESC, tcga_cesc] Dead NaN \n",
"\n",
" cause_of_death \n",
"0 None \n",
"1 None \n",
"2 None \n",
"3 None \n",
"4 None \n",
".. ... \n",
"95 None \n",
"96 None \n",
"97 None \n",
"98 None \n",
"99 None \n",
"\n",
"[2609 rows x 11 columns]"
]
},
"execution_count": 21,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"subsdf # view the subject dataframe"
]
},
{
"cell_type": "markdown",
"id": "75bcbe86",
"metadata": {},
"source": [
"## Merging Results across Endpoints\n",
"\n",
"\n",
" \n",
"Then Julia uses the `subject_id` and `id` fields in each result to merge them together into one big dataset. She also specifies that any columns that are in both tables should be kept and have a suffix added to their name. This will help her to check that her merge worked correctly:"
]
},
{
"cell_type": "code",
"execution_count": 22,
"id": "9b7a3383",
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"\n",
" \n",
" | \n",
" researchsubject_id | \n",
" researchsubject_identifier | \n",
" member_of_research_project | \n",
" primary_diagnosis_condition | \n",
" primary_diagnosis_site | \n",
" subject_id | \n",
" subject_identifier | \n",
" species | \n",
" sex | \n",
" race | \n",
" ethnicity | \n",
" days_to_birth | \n",
" subject_associated_project | \n",
" vital_status | \n",
" days_to_death | \n",
" cause_of_death | \n",
"
\n",
" | Loading... (need help?) |
\n",
"\n",
"\n"
],
"text/plain": [
" researchsubject_id \\\n",
"0 d6929fe7-908d-4684-b799-1c7d1bfe4835 \n",
"1 e9de2706-246c-46ae-a48c-7cb9bff7e723 \n",
"2 0cabe364-1e30-433d-b301-3fcffd953b09 \n",
"3 14c744c0-1ae8-4d8b-bb7c-6149019dc1a3 \n",
"4 164241df-c75b-482d-ad6a-84b0ee9aaf66 \n",
"... ... \n",
"3192 829273bb-eb58-4c6f-b192-599344f5b1ed \n",
"3193 92086136-3281-4ce9-a2df-4b597aaf22c8 \n",
"3194 C3N-01766__cptac_ucec \n",
"3195 TCGA-AX-A2H7__tcga_ucec \n",
"3196 a307e88b-745c-469e-ba24-a7cc544e8859 \n",
"\n",
" researchsubject_identifier \\\n",
"0 [{'system': 'GDC', 'value': 'd6929fe7-908d-468... \n",
"1 [{'system': 'GDC', 'value': 'e9de2706-246c-46a... \n",
"2 [{'system': 'GDC', 'value': '0cabe364-1e30-433... \n",
"3 [{'system': 'GDC', 'value': '14c744c0-1ae8-4d8... \n",
"4 [{'system': 'GDC', 'value': '164241df-c75b-482... \n",
"... ... \n",
"3192 [{'system': 'GDC', 'value': '829273bb-eb58-4c6... \n",
"3193 [{'system': 'GDC', 'value': '92086136-3281-4ce... \n",
"3194 [{'system': 'IDC', 'value': 'C3N-01766__cptac_... \n",
"3195 [{'system': 'IDC', 'value': 'TCGA-AX-A2H7__tcg... \n",
"3196 [{'system': 'GDC', 'value': 'a307e88b-745c-469... \n",
"\n",
" member_of_research_project primary_diagnosis_condition \\\n",
"0 GENIE-MSK Myomatous Neoplasms \n",
"1 GENIE-DFCI Epithelial Neoplasms, NOS \n",
"2 FM-AD Cystic, Mucinous and Serous Neoplasms \n",
"3 GENIE-UHN Epithelial Neoplasms, NOS \n",
"4 GENIE-DFCI Complex Mixed and Stromal Neoplasms \n",
"... ... ... \n",
"3192 GENIE-MSK Complex Epithelial Neoplasms \n",
"3193 GENIE-NKI Cystic, Mucinous and Serous Neoplasms \n",
"3194 cptac_ucec None \n",
"3195 tcga_ucec None \n",
"3196 GENIE-DFCI Complex Mixed and Stromal Neoplasms \n",
"\n",
" primary_diagnosis_site subject_id \\\n",
"0 Uterus, NOS GENIE-MSK-P-0003754 \n",
"1 Corpus uteri GENIE-DFCI-036994 \n",
"2 Uterus, NOS AD10783 \n",
"3 Corpus uteri GENIE-UHN-944776 \n",
"4 Corpus uteri GENIE-DFCI-009119 \n",
"... ... ... \n",
"3192 Cervix uteri GENIE-MSK-P-0008316 \n",
"3193 Cervix uteri GENIE-NKI-3GI3 \n",
"3194 Uterus C3N-01766 \n",
"3195 Uterus TCGA-AX-A2H7 \n",
"3196 Corpus uteri GENIE-DFCI-023832 \n",
"\n",
" subject_identifier species sex \\\n",
"0 [{'system': 'GDC', 'value': 'GENIE-MSK-P-00037... homo sapiens female \n",
"1 [{'system': 'GDC', 'value': 'GENIE-DFCI-036994'}] homo sapiens female \n",
"2 [{'system': 'GDC', 'value': 'AD10783'}] homo sapiens female \n",
"3 [{'system': 'GDC', 'value': 'GENIE-UHN-944776'}] homo sapiens female \n",
"4 [{'system': 'GDC', 'value': 'GENIE-DFCI-009119'}] homo sapiens female \n",
"... ... ... ... \n",
"3192 [{'system': 'GDC', 'value': 'GENIE-MSK-P-00083... homo sapiens female \n",
"3193 [{'system': 'GDC', 'value': 'GENIE-NKI-3GI3'}] homo sapiens male \n",
"3194 [{'system': 'IDC', 'value': 'C3N-01766'}] homo sapiens None \n",
"3195 [{'system': 'GDC', 'value': 'TCGA-AX-A2H7'}, {... homo sapiens female \n",
"3196 [{'system': 'GDC', 'value': 'GENIE-DFCI-023832'}] homo sapiens female \n",
"\n",
" race ethnicity days_to_birth \\\n",
"0 white not hispanic or latino NaN \n",
"1 white not hispanic or latino NaN \n",
"2 not reported not reported NaN \n",
"3 asian not allowed to collect NaN \n",
"4 Unknown not hispanic or latino NaN \n",
"... ... ... ... \n",
"3192 white not hispanic or latino NaN \n",
"3193 Unknown not reported NaN \n",
"3194 None None NaN \n",
"3195 white not hispanic or latino NaN \n",
"3196 white not hispanic or latino NaN \n",
"\n",
" subject_associated_project vital_status days_to_death cause_of_death \n",
"0 [GENIE-MSK] Not Reported None None \n",
"1 [GENIE-DFCI] Not Reported None None \n",
"2 [FM-AD] Not Reported NaN None \n",
"3 [GENIE-UHN] Not Reported NaN None \n",
"4 [GENIE-DFCI] Not Reported NaN None \n",
"... ... ... ... ... \n",
"3192 [GENIE-MSK] Not Reported None None \n",
"3193 [GENIE-NKI] Not Reported None None \n",
"3194 [cptac_ucec] None None None \n",
"3195 [TCGA-UCEC, tcga_ucec] Dead NaN None \n",
"3196 [GENIE-DFCI] Not Reported NaN None \n",
"\n",
"[3197 rows x 16 columns]"
]
},
"execution_count": 22,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"allmetadata = pd.merge(rsdf,\n",
" subsdf,\n",
" left_on=\"subject_id\",\n",
" right_on='subject_id',\n",
" suffixes=(\"_rs\", \"_sub\"))\n",
"\n",
"allmetadata"
]
},
{
"cell_type": "markdown",
"id": "ff2e053e",
"metadata": {},
"source": [
"\n",
" \n",
"`subject_id` from the research subject results seems to perfectly match the id data from the subject table, `id_sub`. Julia then checks to see that her dataframe is the right size. She had 3197 researchsubject rows, so she expects 3197 rows here as well:"
]
},
{
"cell_type": "code",
"execution_count": 23,
"id": "e5874a36",
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"\n",
" \n",
" | \n",
" 0 | \n",
"
\n",
" | Loading... (need help?) |
\n",
"\n",
"\n"
],
"text/plain": [
"researchsubject_id 3197\n",
"researchsubject_identifier 3197\n",
"member_of_research_project 3197\n",
"primary_diagnosis_condition 2022\n",
"primary_diagnosis_site 3197\n",
"subject_id 3197\n",
"subject_identifier 3197\n",
"species 3197\n",
"sex 3025\n",
"race 3025\n",
"ethnicity 3025\n",
"days_to_birth 206\n",
"subject_associated_project 3197\n",
"vital_status 3025\n",
"days_to_death 22\n",
"cause_of_death 206\n",
"dtype: int64"
]
},
"execution_count": 23,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"allmetadata.count()"
]
},
{
"cell_type": "markdown",
"id": "024da831",
"metadata": {},
"source": [
"\n",
" \n",
"Satisfied with her results, Julia saves the data out to a csv so she can browse it with Excel:"
]
},
{
"cell_type": "code",
"execution_count": 24,
"id": "b6628de4",
"metadata": {},
"outputs": [],
"source": [
"allmetadata.to_csv(\"allmetadata.csv\")"
]
},
{
"cell_type": "markdown",
"id": "246644d3",
"metadata": {},
"source": [
"\n",
" \n",
"Julia knows from her subject count summary that there are more than 200,000 files associated with her subjects, which is likely far more than she needs. To help her decide what files she wants, Julia uses endpoint chaining to get summary information about the files that are assigned to researchsubjects for her search criteria:\n"
]
},
{
"cell_type": "code",
"execution_count": 25,
"id": "ae1ae079",
"metadata": {
"scrolled": true
},
"outputs": [
{
"data": {
"text/html": [
"Getting results from database\n",
"\n",
"\n"
],
"text/plain": [
"Getting results from database\n",
"\n"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
"\n",
" Total execution time: 0\n",
" min 7.554 sec 7554 ms\n",
" \n",
"\n"
],
"text/plain": [
"\n",
" Total execution time: \u001b[1;36m0\u001b[0m\n",
" min \u001b[1;36m7.554\u001b[0m sec \u001b[1;36m7554\u001b[0m ms\n",
" \n"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
" total : 297379 \n",
"\n"
],
"text/plain": [
" total : \u001b[1;36m297379\u001b[0m \n"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
"\n",
"\n",
" \n",
" \n",
" | data_category | \n",
" count | \n",
"
\n",
" \n",
" \n",
" \n",
" | Imaging | \n",
" 265303 | \n",
"
\n",
" \n",
" | Transcriptome Profiling | \n",
" 2820 | \n",
"
\n",
" \n",
" | Biospecimen | \n",
" 2866 | \n",
"
\n",
" \n",
" | Peptide Spectral Matches | \n",
" 1280 | \n",
"
\n",
" \n",
" | Processed Mass Spectra | \n",
" 640 | \n",
"
\n",
" \n",
" | Simple Nucleotide Variation | \n",
" 9987 | \n",
"
\n",
" \n",
" | Raw Mass Spectra | \n",
" 640 | \n",
"
\n",
" \n",
" | Copy Number Variation | \n",
" 4079 | \n",
"
\n",
" \n",
" | Structural Variation | \n",
" 2192 | \n",
"
\n",
" \n",
" | Clinical | \n",
" 774 | \n",
"
\n",
" \n",
" | DNA Methylation | \n",
" 1947 | \n",
"
\n",
" \n",
" | Sequencing Reads | \n",
" 4142 | \n",
"
\n",
" \n",
" | Somatic Structural Variation | \n",
" 370 | \n",
"
\n",
" \n",
" | Proteome Profiling | \n",
" 339 | \n",
"
\n",
" \n",
"
\n",
"\n",
"\n",
" \n",
" \n",
" | data_type | \n",
" count | \n",
"
\n",
" \n",
" \n",
" \n",
" | None | \n",
" 265303 | \n",
"
\n",
" \n",
" | Proprietary | \n",
" 640 | \n",
"
\n",
" \n",
" | Open Standard | \n",
" 1280 | \n",
"
\n",
" \n",
" | Masked Annotated Somatic Mutation | \n",
" 1024 | \n",
"
\n",
" \n",
" | Transcript Fusion | \n",
" 2278 | \n",
"
\n",
" \n",
" | Annotated Somatic Mutation | \n",
" 4871 | \n",
"
\n",
" \n",
" | Text | \n",
" 640 | \n",
"
\n",
" \n",
" | Masked Copy Number Segment | \n",
" 998 | \n",
"
\n",
" \n",
" | Gene Level Copy Number | \n",
" 637 | \n",
"
\n",
" \n",
" | Slide Image | \n",
" 1111 | \n",
"
\n",
" \n",
" | Aligned Reads | \n",
" 4142 | \n",
"
\n",
" \n",
" | Biospecimen Supplement | \n",
" 1755 | \n",
"
\n",
" \n",
" | Structural Rearrangement | \n",
" 284 | \n",
"
\n",
" \n",
" | Clinical Supplement | \n",
" 773 | \n",
"
\n",
" \n",
" | Raw Simple Somatic Mutation | \n",
" 2811 | \n",
"
\n",
" \n",
" | Masked Intensities | \n",
" 1298 | \n",
"
\n",
" \n",
" | Copy Number Segment | \n",
" 1140 | \n",
"
\n",
" \n",
" | Aggregated Somatic Mutation | \n",
" 732 | \n",
"
\n",
" \n",
" | Isoform Expression Quantification | \n",
" 700 | \n",
"
\n",
" \n",
" | Methylation Beta Value | \n",
" 649 | \n",
"
\n",
" \n",
" | Gene Expression Quantification | \n",
" 710 | \n",
"
\n",
" \n",
" | Gene Level Copy Number Scores | \n",
" 809 | \n",
"
\n",
" \n",
" | Protein Expression Quantification | \n",
" 339 | \n",
"
\n",
" \n",
" | Splice Junction Quantification | \n",
" 710 | \n",
"
\n",
" \n",
" | miRNA Expression Quantification | \n",
" 700 | \n",
"
\n",
" \n",
" | Allele-specific Copy Number Segment | \n",
" 495 | \n",
"
\n",
" \n",
" | Masked Somatic Mutation | \n",
" 549 | \n",
"
\n",
" \n",
" | Pathology Report | \n",
" 1 | \n",
"
\n",
" \n",
"
\n",
"\n",
"\n",
" \n",
" \n",
" | file_format | \n",
" count | \n",
"
\n",
" \n",
" \n",
" \n",
" | DICOM | \n",
" 265303 | \n",
"
\n",
" \n",
" | vendor-specific | \n",
" 640 | \n",
"
\n",
" \n",
" | BAM | \n",
" 4142 | \n",
"
\n",
" \n",
" | VCF | \n",
" 5480 | \n",
"
\n",
" \n",
" | tsv | \n",
" 640 | \n",
"
\n",
" \n",
" | BCR Biotab | \n",
" 76 | \n",
"
\n",
" \n",
" | TSV | \n",
" 3980 | \n",
"
\n",
" \n",
" | mzML | \n",
" 640 | \n",
"
\n",
" \n",
" | TXT | \n",
" 4827 | \n",
"
\n",
" \n",
" | BCR SSF XML | \n",
" 517 | \n",
"
\n",
" \n",
" | MAF | \n",
" 4649 | \n",
"
\n",
" \n",
" | SVS | \n",
" 1111 | \n",
"
\n",
" \n",
" | BEDPE | \n",
" 1504 | \n",
"
\n",
" \n",
" | mzIdentML | \n",
" 640 | \n",
"
\n",
" \n",
" | BCR XML | \n",
" 1217 | \n",
"
\n",
" \n",
" | BCR PPS XML | \n",
" 193 | \n",
"
\n",
" \n",
" | IDAT | \n",
" 1298 | \n",
"
\n",
" \n",
" | CDC JSON | \n",
" 1 | \n",
"
\n",
" \n",
" | BCR Auxiliary XML | \n",
" 474 | \n",
"
\n",
" \n",
" | BCR OMF XML | \n",
" 39 | \n",
"
\n",
" \n",
" | XLSX | \n",
" 7 | \n",
"
\n",
" \n",
" | PDF | \n",
" 1 | \n",
"
\n",
" \n",
"
\n",
"\n",
"\n",
" \n",
" \n",
" | file_identifier_system | \n",
" count | \n",
"
\n",
" \n",
" \n",
" \n",
" | IDC | \n",
" 265303 | \n",
"
\n",
" \n",
" | GDC | \n",
" 29516 | \n",
"
\n",
" \n",
" | PDC | \n",
" 2560 | \n",
"
\n",
" \n",
"
\n"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/plain": []
},
"execution_count": 25,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"NoAdenoData.researchsubject.file.count.run()"
]
},
{
"cell_type": "markdown",
"id": "111f47c0",
"metadata": {},
"source": [
"\n",
" \n",
"Julia decides that a good place to start would be with Slide Images. There's only 1111, so she should be able to quickly scan through them over the next few days and see if they will be useful. So she adds one more filter on her search:"
]
},
{
"cell_type": "code",
"execution_count": 26,
"id": "d4a170b3",
"metadata": {
"scrolled": true
},
"outputs": [
{
"data": {
"text/html": [
"Getting results from database\n",
"\n",
"\n"
],
"text/plain": [
"Getting results from database\n",
"\n"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
"\n",
" Total execution time: 0\n",
" min 7.392 sec 7392 ms\n",
" \n",
"\n"
],
"text/plain": [
"\n",
" Total execution time: \u001b[1;36m0\u001b[0m\n",
" min \u001b[1;36m7.392\u001b[0m sec \u001b[1;36m7392\u001b[0m ms\n",
" \n"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
" total : 1111 \n",
"\n"
],
"text/plain": [
" total : \u001b[1;36m1111\u001b[0m \n"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
"\n",
"\n",
" \n",
" \n",
" | data_category | \n",
" count | \n",
"
\n",
" \n",
" \n",
" \n",
" | Biospecimen | \n",
" 1111 | \n",
"
\n",
" \n",
"
\n",
"\n",
"\n",
" \n",
" \n",
" | data_type | \n",
" count | \n",
"
\n",
" \n",
" \n",
" \n",
" | Slide Image | \n",
" 1111 | \n",
"
\n",
" \n",
"
\n",
"\n",
"\n",
" \n",
" \n",
" | file_format | \n",
" count | \n",
"
\n",
" \n",
" \n",
" \n",
" | SVS | \n",
" 1111 | \n",
"
\n",
" \n",
"
\n",
"\n",
"\n",
" \n",
" \n",
" | file_identifier_system | \n",
" count | \n",
"
\n",
" \n",
" \n",
" \n",
" | GDC | \n",
" 1111 | \n",
"
\n",
" \n",
"
\n"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/plain": []
},
"execution_count": 26,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"JustSlides = Q('data_type = \"Slide Image\"')\n",
"NoadenoJustSlides = NoAdenoData.AND(JustSlides)\n",
"NoadenoJustSlides.researchsubject.file.count.run()"
]
},
{
"cell_type": "markdown",
"id": "0e385faf",
"metadata": {},
"source": [
"\n",
" \n",
"Finally, Julia uses the pagenation function again to get all the slide files, and merges her metadata file with this file information. This way she will be able to review what phenotypes each slide is associated with:"
]
},
{
"cell_type": "code",
"execution_count": 27,
"id": "f45f6503",
"metadata": {
"scrolled": false
},
"outputs": [
{
"data": {
"text/html": [
"Getting results from database\n",
"\n",
"\n"
],
"text/plain": [
"Getting results from database\n",
"\n"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
"\n",
" Total execution time: 0\n",
" min 7.449 sec 7449 ms\n",
" \n",
"\n"
],
"text/plain": [
"\n",
" Total execution time: \u001b[1;36m0\u001b[0m\n",
" min \u001b[1;36m7.449\u001b[0m sec \u001b[1;36m7449\u001b[0m ms\n",
" \n"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "b7c55c5c15e649fca3674bd8a00f9208",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"Output()"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
"\n"
],
"text/plain": []
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
"\n",
"\n"
],
"text/plain": [
"\n"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
"\n",
"\n"
],
"text/plain": [
"\n"
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"slides = NoadenoJustSlides.researchsubject.file.run()\n",
"slidesdf = slides.auto_paginator(to_df=True)"
]
},
{
"cell_type": "code",
"execution_count": 28,
"id": "0d1f7a21",
"metadata": {
"scrolled": true
},
"outputs": [
{
"data": {
"text/html": [
"\n",
" \n",
" | \n",
" file_id | \n",
" file_identifier | \n",
" label | \n",
" data_category | \n",
" data_type | \n",
" file_format | \n",
" file_associated_project | \n",
" drs_uri | \n",
" byte_size | \n",
" checksum | \n",
" data_modality | \n",
" imaging_modality | \n",
" dbgap_accession_number | \n",
" imaging_series | \n",
" researchsubject_id | \n",
" subject_id | \n",
" researchsubject_identifier | \n",
" member_of_research_project | \n",
" primary_diagnosis_condition | \n",
" primary_diagnosis_site | \n",
" subject_identifier | \n",
" species | \n",
" sex | \n",
" race | \n",
" ethnicity | \n",
" days_to_birth | \n",
" subject_associated_project | \n",
" vital_status | \n",
" days_to_death | \n",
" cause_of_death | \n",
"
\n",
" | Loading... (need help?) |
\n",
"\n",
"\n"
],
"text/plain": [
" file_id \\\n",
"0 79dea52e-0f01-432f-80d2-605d6053052e \n",
"1 7bf22346-d3ab-4ff8-8568-541cbec1470c \n",
"2 3a3f1e4b-9d8c-47d2-91dd-f779f2a36a40 \n",
"3 7e640239-613a-49ec-aa97-e6af328efe93 \n",
"4 a14a9dd7-5c0a-49d7-bc15-4ed2a483fd06 \n",
"... ... \n",
"1106 9bf82894-b9bb-4ed3-a365-cee3e2a56480 \n",
"1107 7a6c0676-f25f-4fd9-a975-d2e4c01c058c \n",
"1108 3c9c7394-a1a7-41e3-b3d7-4b502c6c5b6c \n",
"1109 b97eb4e2-e6af-4f62-a203-4751c183b4df \n",
"1110 a5d047ba-6816-4c12-a971-03bded22f34d \n",
"\n",
" file_identifier \\\n",
"0 [{'system': 'GDC', 'value': '79dea52e-0f01-432... \n",
"1 [{'system': 'GDC', 'value': '7bf22346-d3ab-4ff... \n",
"2 [{'system': 'GDC', 'value': '3a3f1e4b-9d8c-47d... \n",
"3 [{'system': 'GDC', 'value': '7e640239-613a-49e... \n",
"4 [{'system': 'GDC', 'value': 'a14a9dd7-5c0a-49d... \n",
"... ... \n",
"1106 [{'system': 'GDC', 'value': '9bf82894-b9bb-4ed... \n",
"1107 [{'system': 'GDC', 'value': '7a6c0676-f25f-4fd... \n",
"1108 [{'system': 'GDC', 'value': '3c9c7394-a1a7-41e... \n",
"1109 [{'system': 'GDC', 'value': 'b97eb4e2-e6af-4f6... \n",
"1110 [{'system': 'GDC', 'value': 'a5d047ba-6816-4c1... \n",
"\n",
" label data_category \\\n",
"0 TCGA-N6-A4VF-01A-03-TSC.C6837FF0-A19E-4D16-A1B... Biospecimen \n",
"1 TCGA-N6-A4VF-11A-01-TSA.07939203-9432-431E-96D... Biospecimen \n",
"2 TCGA-N6-A4VF-01Z-00-DX1.D78EF5C0-E015-42DB-8D4... Biospecimen \n",
"3 TCGA-QN-A5NN-01Z-00-DX2.0FE8A3DA-ABAD-4517-9D4... Biospecimen \n",
"4 TCGA-QN-A5NN-01Z-00-DX4.AB57D943-2A7C-46E8-AD2... Biospecimen \n",
"... ... ... \n",
"1106 TCGA-C5-A1BK-01B-01-TS1.86c00487-ef71-4272-96c... Biospecimen \n",
"1107 TCGA-C5-A1BK-01Z-00-DX1.B01FAC89-30CC-4CDB-910... Biospecimen \n",
"1108 TCGA-PN-A8MA-01A-01-DX1.978164EB-6260-45BA-A87... Biospecimen \n",
"1109 TCGA-C5-A907-01A-01-TS1.E0D13713-6294-47D5-9D0... Biospecimen \n",
"1110 TCGA-C5-A907-01Z-00-DX1.7DEEFC23-B631-419E-B5E... Biospecimen \n",
"\n",
" data_type file_format file_associated_project \\\n",
"0 Slide Image SVS [TCGA-UCS] \n",
"1 Slide Image SVS [TCGA-UCS] \n",
"2 Slide Image SVS [TCGA-UCS] \n",
"3 Slide Image SVS [TCGA-UCS] \n",
"4 Slide Image SVS [TCGA-UCS] \n",
"... ... ... ... \n",
"1106 Slide Image SVS [TCGA-CESC] \n",
"1107 Slide Image SVS [TCGA-CESC] \n",
"1108 Slide Image SVS [TCGA-CESC] \n",
"1109 Slide Image SVS [TCGA-CESC] \n",
"1110 Slide Image SVS [TCGA-CESC] \n",
"\n",
" drs_uri byte_size \\\n",
"0 drs://dg.4DFC:79dea52e-0f01-432f-80d2-605d6053... 89647451 \n",
"1 drs://dg.4DFC:7bf22346-d3ab-4ff8-8568-541cbec1... 47409767 \n",
"2 drs://dg.4DFC:3a3f1e4b-9d8c-47d2-91dd-f779f2a3... 1465712675 \n",
"3 drs://dg.4DFC:7e640239-613a-49ec-aa97-e6af328e... 1477683125 \n",
"4 drs://dg.4DFC:a14a9dd7-5c0a-49d7-bc15-4ed2a483... 1999093775 \n",
"... ... ... \n",
"1106 drs://dg.4DFC:9bf82894-b9bb-4ed3-a365-cee3e2a5... 175306257 \n",
"1107 drs://dg.4DFC:7a6c0676-f25f-4fd9-a975-d2e4c01c... 1589829656 \n",
"1108 drs://dg.4DFC:3c9c7394-a1a7-41e3-b3d7-4b502c6c... 88338500 \n",
"1109 drs://dg.4DFC:b97eb4e2-e6af-4f62-a203-4751c183... 257797557 \n",
"1110 drs://dg.4DFC:a5d047ba-6816-4c12-a971-03bded22... 300025421 \n",
"\n",
" checksum ... \\\n",
"0 ffb1321391581951ad03a5bd18f7e0b4 ... \n",
"1 aafe80885eaeaf41569e7b09cf950edd ... \n",
"2 b0feb735368ad896c369aabad27873b9 ... \n",
"3 3936fa9a7ad9924d74abf2627f116694 ... \n",
"4 b6b33e61eaba4c0bc787f27638986c8b ... \n",
"... ... ... \n",
"1106 e92d388fc448d48b43b5bc5d267536f3 ... \n",
"1107 8ca27e30c738192a9b65b546b50c9e5a ... \n",
"1108 c99a39e8eaf4c845db3896cb664de33e ... \n",
"1109 abcc25a2e40c4b9b7891e19f38d3ffeb ... \n",
"1110 9d54b624e6a01c004ade2aba551742c0 ... \n",
"\n",
" subject_identifier species sex \\\n",
"0 [{'system': 'GDC', 'value': 'TCGA-N6-A4VF'}, {... homo sapiens female \n",
"1 [{'system': 'GDC', 'value': 'TCGA-N6-A4VF'}, {... homo sapiens female \n",
"2 [{'system': 'GDC', 'value': 'TCGA-N6-A4VF'}, {... homo sapiens female \n",
"3 [{'system': 'GDC', 'value': 'TCGA-QN-A5NN'}, {... homo sapiens female \n",
"4 [{'system': 'GDC', 'value': 'TCGA-QN-A5NN'}, {... homo sapiens female \n",
"... ... ... ... \n",
"1106 [{'system': 'GDC', 'value': 'TCGA-C5-A1BK'}, {... homo sapiens female \n",
"1107 [{'system': 'GDC', 'value': 'TCGA-C5-A1BK'}, {... homo sapiens female \n",
"1108 [{'system': 'GDC', 'value': 'TCGA-PN-A8MA'}, {... homo sapiens female \n",
"1109 [{'system': 'GDC', 'value': 'TCGA-C5-A907'}, {... homo sapiens female \n",
"1110 [{'system': 'GDC', 'value': 'TCGA-C5-A907'}, {... homo sapiens female \n",
"\n",
" race ethnicity days_to_birth \\\n",
"0 white not hispanic or latino NaN \n",
"1 white not hispanic or latino NaN \n",
"2 white not hispanic or latino NaN \n",
"3 asian not hispanic or latino NaN \n",
"4 asian not hispanic or latino NaN \n",
"... ... ... ... \n",
"1106 black or african american not hispanic or latino NaN \n",
"1107 black or african american not hispanic or latino NaN \n",
"1108 black or african american not hispanic or latino NaN \n",
"1109 white not hispanic or latino NaN \n",
"1110 white not hispanic or latino NaN \n",
"\n",
" subject_associated_project vital_status days_to_death cause_of_death \n",
"0 [TCGA-UCS, tcga_ucs] Dead NaN None \n",
"1 [TCGA-UCS, tcga_ucs] Dead NaN None \n",
"2 [TCGA-UCS, tcga_ucs] Dead NaN None \n",
"3 [TCGA-UCS, tcga_ucs] Alive None None \n",
"4 [TCGA-UCS, tcga_ucs] Alive None None \n",
"... ... ... ... ... \n",
"1106 [TCGA-CESC, tcga_cesc] Alive NaN None \n",
"1107 [TCGA-CESC, tcga_cesc] Alive NaN None \n",
"1108 [TCGA-CESC, tcga_cesc] Alive None None \n",
"1109 [TCGA-CESC, tcga_cesc] Alive NaN None \n",
"1110 [TCGA-CESC, tcga_cesc] Alive NaN None \n",
"\n",
"[1111 rows x 30 columns]"
]
},
"execution_count": 28,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"slidemetadata = pd.merge(slidesdf, \n",
" allmetadata, \n",
" left_on=(\"subject_id\",\"researchsubject_id\"),\n",
" right_on=(\"subject_id\", \"researchsubject_id\"),\n",
" suffixes=(\"_slide\", \"_all\"))\n",
"slidemetadata"
]
},
{
"cell_type": "code",
"execution_count": 29,
"id": "306e479c",
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"\n",
" \n",
" | \n",
" 0 | \n",
"
\n",
" | Loading... (need help?) |
\n",
"\n",
"\n"
],
"text/plain": [
"file_id 1111\n",
"file_identifier 1111\n",
"label 1111\n",
"data_category 1111\n",
"data_type 1111\n",
"file_format 1111\n",
"file_associated_project 1111\n",
"drs_uri 1111\n",
"byte_size 1111\n",
"checksum 1111\n",
"data_modality 1111\n",
"imaging_modality 0\n",
"dbgap_accession_number 0\n",
"imaging_series 0\n",
"researchsubject_id 1111\n",
"subject_id 1111\n",
"researchsubject_identifier 1111\n",
"member_of_research_project 1111\n",
"primary_diagnosis_condition 1111\n",
"primary_diagnosis_site 1111\n",
"subject_identifier 1111\n",
"species 1111\n",
"sex 1099\n",
"race 1099\n",
"ethnicity 1099\n",
"days_to_birth 0\n",
"subject_associated_project 1111\n",
"vital_status 1099\n",
"days_to_death 0\n",
"cause_of_death 0\n",
"dtype: int64"
]
},
"execution_count": 29,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"slidemetadata.count()"
]
},
{
"cell_type": "markdown",
"id": "369a3068",
"metadata": {},
"source": [
"\n",
" \n",
"Julia saves this dataframe to a csv as well, and now she has all the information she needs to begin work on her project. She can use the `drs_id` column information to directly download the images she is interested in using a DRS resolver, or she can input the DRS IDs at a cloud workspace such as [Terra](https://terra.bio/) or the [Cancer Genomics Cloud](https://www.cancergenomicscloud.org/) to view the images online. In either case, she has all the metadata she needs to get started, and can save this notebook of her work in case she'd like to come back and modify her search."
]
},
{
"cell_type": "code",
"execution_count": 30,
"id": "8ef6e0fb",
"metadata": {},
"outputs": [],
"source": [
"slidemetadata.to_csv(\"slidemetadata.csv\")"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.10.6"
}
},
"nbformat": 4,
"nbformat_minor": 5
}