Introduction to the Nomis API

nomisr is for accessing Nomis data with R. The Nomis API is free to access without registration, and contains up-to-date official statistics, including data from the Census, the Labour Force Survey, DWP benefit statistics and other economic and demographic data. Nomis is maintained on behalf of the Office for National Statistics by the University of Durham.

There is a lot of data available through Nomis, and there are some limits to the amount of data that can be retrieved within a certain period of time, although those are not published. For more details, see the full API documentation from Nomis.

Nomis data is based around administrative and statistical geographies, and a particular geography should be specified when downloading data.

nomisr is designed around a pipeline of three key functions: nomis_data_info(), nomis_get_metadata() and nomis_get_data(). The nomis_overview(), nomis_content_type() and nomis_search() functions can assist with this.

Querying data availability

The nomis_data_info() function is focused on the structure and coverage of the available datasets.

Use the nomis_data_info() function without any parameters to get a tibble with metadata for all available datasets:

## # A tibble: 6 x 14
##   agencyid id    uri   version annotations.ann… components.attr…
##   <chr>    <chr> <chr>   <dbl> <list>           <list>          
## 1 NOMIS    NM_1… Nm-1…       1 <data.frame [10… <data.frame [7 …
## 2 NOMIS    NM_2… Nm-2…       1 <data.frame [6 … <data.frame [7 …
## 3 NOMIS    NM_4… Nm-4…       1 <data.frame [8 … <data.frame [7 …
## 4 NOMIS    NM_5… Nm-5…       1 <data.frame [5 … <data.frame [7 …
## 5 NOMIS    NM_6… Nm-6…       1 <data.frame [6 … <data.frame [7 …
## 6 NOMIS    NM_7… Nm-7…       1 <data.frame [6 … <data.frame [7 …
## # ... with 8 more variables: components.dimension <list>,
## #   components.primarymeasure.conceptref <chr>,
## #   components.timedimension.codelist <chr>,
## #   components.timedimension.conceptref <chr>, description.value <chr>,
## #   description.lang <chr>, name.value <chr>, name.lang <chr>

nomis_data_info() can also be used to query metadata from a specific dataset, using its ID. The example below uses the “LC4408EW - Tenure by number of persons per bedroom in household by household type” dataset from the 2011 census, which has the ID “NM_893_1”.

## Observations: 1
## Variables: 12
## $ agencyid                             <chr> "NOMIS"
## $ id                                   <chr> "NM_893_1"
## $ uri                                  <chr> "Nm-893d1"
## $ version                              <dbl> 1
## $ annotations.annotation               <list> [<c("Current (being acti...
## $ components.attribute                 <list> [<c("Mandatory", "Condit...
## $ components.dimension                 <list> [<c("CL_893_1_GEOGRAPHY"...
## $ components.primarymeasure.conceptref <chr> "OBS_VALUE"
## $ components.timedimension.codelist    <chr> "CL_893_1_TIME"
## $ components.timedimension.conceptref  <chr> "TIME"
## $ name.value                           <chr> "LC4408EW - Tenure by num...
## $ name.lang                            <chr> "en"

When a tibble with metadata for all datasets or a specific dataset is returned, three of the columns, annotations.annotation, components.attribute and components.dimension, are list-columns of data frames. annotations.annotation contains metadata on the dataset, including units and current status. components.attribute contains more detailed status metadata. components.dimension contains the grouping and summary variables available in the dataset, which vary between different datasets.

The example below shows how to access data stored in list columns returned from the Nomis API. In the case of requests for metadata from a single dataset, the three columns are all lists with a length of 1. If requesting all dataset information with nomis_data_info(), each row is a list of length 1. Each list contains a data.frame, of varrying dimensions depending on the column and dataset. You can unnest individual list-columns to display their data in the same row as data from the rest of the tibble. Due to the differing lengths of the list-columns returned by nomis_data_info(), only one list-column can be unnested at a time.

## [1] "list"
## [1] 1
## [1] "data.frame"
y %>% pull(annotations.annotation) %>% class()
## [1] "list"
y %>% pull(annotations.annotation) %>% .[[1]] %>% class()
## [1] "data.frame"
y %>% pull(annotations.annotation) %>% purrr::pluck() %>% class()
## [1] "list"
## Observations: 14
## Variables: 11
## $ agencyid                             <chr> "NOMIS", "NOMIS", "NOMIS"...
## $ id                                   <chr> "NM_893_1", "NM_893_1", "...
## $ uri                                  <chr> "Nm-893d1", "Nm-893d1", "...
## $ version                              <dbl> 1, 1, 1, 1, 1, 1, 1, 1, 1...
## $ components.primarymeasure.conceptref <chr> "OBS_VALUE", "OBS_VALUE",...
## $ components.timedimension.codelist    <chr> "CL_893_1_TIME", "CL_893_...
## $ components.timedimension.conceptref  <chr> "TIME", "TIME", "TIME", "...
## $ name.value                           <chr> "LC4408EW - Tenure by num...
## $ name.lang                            <chr> "en", "en", "en", "en", "...
## $ annotationtext                       <chr> "Current (being actively ...
## $ annotationtitle                      <chr> "Status", "Keywords", "Un...

Searching for data

nomisr also contains the nomis_search() function to search for datasets on particular topics. nomis_search() can be used to search in one or more of dataset names, descriptions, keywords, content type and units. If using multiple parameters, nomis_search() will return information on all datasets that match one or more parameters. Character vectors of strings can be used in searches, and likewise nomis_search() will return information on datasets that match one or more queries. The * is used as a wildcard symbol. nomis_search() returns metadata in the same format as nomis_data_info(), including using list-columns. The nomis_content_type() function can assist in identifying content type IDs for nomis_search().

a <- nomis_search(name = '*jobseekers*', keywords = 'Claimants')

tibble::glimpse(a)
## Observations: 33
## Variables: 14
## $ agencyid                             <chr> "NOMIS", "NOMIS", "NOMIS"...
## $ id                                   <chr> "NM_1_1", "NM_2_1", "NM_6...
## $ uri                                  <chr> "Nm-1d1", "Nm-2d1", "Nm-6...
## $ version                              <dbl> 1, 1, 1, 1, 1, 1, 1, 1, 1...
## $ annotations.annotation               <list> [<c("Current (being acti...
## $ components.attribute                 <list> [<c("Mandatory", "Condit...
## $ components.dimension                 <list> [<c("CL_1_1_GEOGRAPHY", ...
## $ components.primarymeasure.conceptref <chr> "OBS_VALUE", "OBS_VALUE",...
## $ components.timedimension.codelist    <chr> "CL_1_1_TIME", "CL_2_1_TI...
## $ components.timedimension.conceptref  <chr> "TIME", "TIME", "TIME", "...
## $ description.value                    <chr> "Records the number of pe...
## $ description.lang                     <chr> "en", "en", "en", "en", N...
## $ name.value                           <chr> "Jobseeker's Allowance wi...
## $ name.lang                            <chr> "en", "en", "en", "en", "...
a %>% tidyr::unnest(components.attribute) %>% glimpse()
## Observations: 231
## Variables: 15
## $ agencyid                             <chr> "NOMIS", "NOMIS", "NOMIS"...
## $ id                                   <chr> "NM_1_1", "NM_1_1", "NM_1...
## $ uri                                  <chr> "Nm-1d1", "Nm-1d1", "Nm-1...
## $ version                              <dbl> 1, 1, 1, 1, 1, 1, 1, 1, 1...
## $ components.primarymeasure.conceptref <chr> "OBS_VALUE", "OBS_VALUE",...
## $ components.timedimension.codelist    <chr> "CL_1_1_TIME", "CL_1_1_TI...
## $ components.timedimension.conceptref  <chr> "TIME", "TIME", "TIME", "...
## $ description.value                    <chr> "Records the number of pe...
## $ description.lang                     <chr> "en", "en", "en", "en", "...
## $ name.value                           <chr> "Jobseeker's Allowance wi...
## $ name.lang                            <chr> "en", "en", "en", "en", "...
## $ assignmentstatus                     <chr> "Mandatory", "Conditional...
## $ attachmentlevel                      <chr> "Observation", "Observati...
## $ codelist                             <chr> "CL_OBS_STATUS", "CL_OBS_...
## $ conceptref                           <chr> "OBS_STATUS", "OBS_CONF",...
b <- nomis_search(keywords = c('Claimants', '*Year*'))

tibble::glimpse(b)
## Observations: 82
## Variables: 14
## $ agencyid                             <chr> "NOMIS", "NOMIS", "NOMIS"...
## $ id                                   <chr> "NM_1_1", "NM_2_1", "NM_5...
## $ uri                                  <chr> "Nm-1d1", "Nm-2d1", "Nm-5...
## $ version                              <dbl> 1, 1, 1, 1, 1, 1, 1, 1, 1...
## $ annotations.annotation               <list> [<c("Current (being acti...
## $ components.attribute                 <list> [<c("Mandatory", "Condit...
## $ components.dimension                 <list> [<c("CL_1_1_GEOGRAPHY", ...
## $ components.primarymeasure.conceptref <chr> "OBS_VALUE", "OBS_VALUE",...
## $ components.timedimension.codelist    <chr> "CL_1_1_TIME", "CL_2_1_TI...
## $ components.timedimension.conceptref  <chr> "TIME", "TIME", "TIME", "...
## $ description.value                    <chr> "Records the number of pe...
## $ description.lang                     <chr> "en", "en", "en", "en", "...
## $ name.value                           <chr> "Jobseeker's Allowance wi...
## $ name.lang                            <chr> "en", "en", "en", "en", "...
b %>% tidyr::unnest(components.attribute) %>% glimpse()
## Observations: 574
## Variables: 15
## $ agencyid                             <chr> "NOMIS", "NOMIS", "NOMIS"...
## $ id                                   <chr> "NM_1_1", "NM_1_1", "NM_1...
## $ uri                                  <chr> "Nm-1d1", "Nm-1d1", "Nm-1...
## $ version                              <dbl> 1, 1, 1, 1, 1, 1, 1, 1, 1...
## $ components.primarymeasure.conceptref <chr> "OBS_VALUE", "OBS_VALUE",...
## $ components.timedimension.codelist    <chr> "CL_1_1_TIME", "CL_1_1_TI...
## $ components.timedimension.conceptref  <chr> "TIME", "TIME", "TIME", "...
## $ description.value                    <chr> "Records the number of pe...
## $ description.lang                     <chr> "en", "en", "en", "en", "...
## $ name.value                           <chr> "Jobseeker's Allowance wi...
## $ name.lang                            <chr> "en", "en", "en", "en", "...
## $ assignmentstatus                     <chr> "Mandatory", "Conditional...
## $ attachmentlevel                      <chr> "Observation", "Observati...
## $ codelist                             <chr> "CL_OBS_STATUS", "CL_OBS_...
## $ conceptref                           <chr> "OBS_STATUS", "OBS_CONF",...

Other ways to access metadata

nomis_overview() returns a tibble with a generalised overview of a given dataset.

## Observations: 20
## Variables: 2
## $ value <list> [[<c("1", "4", "5"), c("NM_1650_1", "NM_1650_4", "NM_16...
## $ name  <chr> "analyses", "analysisname", "analysisnumber", "contact",...

nomis_overview() has a select parameter that can be used to select only particular elements of the overview to return.

 s <- nomis_overview("NM_1650_1", select = c("units", "keywords"))
 
 s %>% tidyr::unnest(name) %>% glimpse()
## Observations: 3
## Variables: 2
## $ value <list> ["NM_1650_1", "Year last worked", [["Persons"]]]
## $ name  <chr> "id", "keywords", "units"

Querying data variables

Vast amounts of data are available through Nomis and so to avoid overwhelming the API, it is good practice to query what concepts are available, using nomis_get_metadata(). While the other metadata functions can return concept metadata, nomis_get_metadata() provides greater flexibility and specificity over the returned metadata than nomis_overview() and nomis_data_info().

The example below queries some of the metadata available through the API for the “LC4408EW - Tenure bynumber of persons per bedroom in household by household type” dataset.

Getting concepts

If provided with just a dataset ID, nomis_get_metadata() will return the concepts available for the given dataset.

## # A tibble: 6 x 3
##   codelist                conceptref     isfrequencydimension
## * <chr>                   <chr>          <chr>               
## 1 CL_893_1_GEOGRAPHY      GEOGRAPHY      false               
## 2 CL_893_1_C_TENHUK11     C_TENHUK11     false               
## 3 CL_893_1_C_PPBROOMHEW11 C_PPBROOMHEW11 false               
## 4 CL_893_1_C_AHTHUK11     C_AHTHUK11     false               
## 5 CL_893_1_MEASURES       MEASURES       false               
## 6 CL_893_1_FREQ           FREQ           true

Concept Values

If provided with a concept name it returns the available values for that concept. However, in some cases, espescially with the geography concept, there are multiple options available, which Nomis labels types. In that case nomis_get_metadata() returns the values of the lowest indexed type available.

  b <- nomis_get_metadata(id = "NM_893_1", concept = "GEOGRAPHY")

  print(b)
## # A tibble: 3 x 4
##   id         parentCode label.en          description.en   
##   <chr>      <chr>      <chr>             <chr>            
## 1 2092957703 <NA>       England and Wales England and Wales
## 2 2092957699 <NA>       England           England          
## 3 2092957700 2092957700 Wales             Wales

We can now pass a generic “type” string to the type parameter in nomis_get_metadata(), which returns all available geography types for dataset “NM_893_1”.

  c <- nomis_get_metadata(id = "NM_893_1", concept = "geography", type = "type")

  print(c)
## # A tibble: 22 x 3
##    id      label.en                       description.en                  
##    <chr>   <chr>                          <chr>                           
##  1 TYPE265 NHS area teams                 NHS area teams                  
##  2 TYPE266 clinical commissioning groups  clinical commissioning groups   
##  3 TYPE267 built-up areas including subd… built-up areas including subdiv…
##  4 TYPE269 built-up areas                 built-up areas                  
##  5 TYPE273 national assembly for wales e… national assembly for wales ele…
##  6 TYPE274 postcode areas                 postcode areas                  
##  7 TYPE275 postcode districts             postcode districts              
##  8 TYPE276 postcode sectors               postcode sectors                
##  9 TYPE277 national assembly for wales c… national assembly for wales con…
## 10 TYPE279 parishes 2011                  parishes 2011                   
## # ... with 12 more rows

Passing a specific type to the type parameter, in this case “TYPE460” for all post-2010 parliamentary constituencies, returns a tibble with geographic codes for those specific constituencies, which can be used to filter queries.

  d <- nomis_get_metadata(id = "NM_893_1", concept = "geography", type = "TYPE460")

  print(d)
## # A tibble: 573 x 4
##    id         parentCode label.en           description.en    
##    <chr>      <chr>      <chr>              <chr>             
##  1 1929379865 2013265921 Berwick-upon-Tweed Berwick-upon-Tweed
##  2 1929379880 2013265921 Bishop Auckland    Bishop Auckland   
##  3 1929379885 2013265921 Blaydon            Blaydon           
##  4 1929379886 2013265921 Blyth Valley       Blyth Valley      
##  5 1929379952 2013265921 City of Durham     City of Durham    
##  6 1929379969 2013265921 Darlington         Darlington        
##  7 1929379988 2013265921 Easington          Easington         
##  8 1929380020 2013265921 Gateshead          Gateshead         
##  9 1929380044 2013265921 Hartlepool         Hartlepool        
## 10 1929380057 2013265921 Hexham             Hexham            
## # ... with 563 more rows

The vast majority (98% as of February 2018) of Nomis datasets include a geographic variable.

Downloading data

Using the information above, we can now query the latest data on bedroom occupancy per household type in different NHS clinical commissioning groups.

z <- nomis_get_data(id = "NM_893_1", time = "latest", geography = "TYPE266")

print(z)
## # A tibble: 44,310 x 40
##    DATE  DATE_NAME DATE_CODE DATE_TYPE DATE_TYPECODE DATE_SORTORDER
##    <chr> <chr>     <chr>     <chr>     <chr>         <chr>         
##  1 2011  2011      2011      date      0             0             
##  2 2011  2011      2011      date      0             0             
##  3 2011  2011      2011      date      0             0             
##  4 2011  2011      2011      date      0             0             
##  5 2011  2011      2011      date      0             0             
##  6 2011  2011      2011      date      0             0             
##  7 2011  2011      2011      date      0             0             
##  8 2011  2011      2011      date      0             0             
##  9 2011  2011      2011      date      0             0             
## 10 2011  2011      2011      date      0             0             
## # ... with 44,300 more rows, and 34 more variables: GEOGRAPHY <chr>,
## #   GEOGRAPHY_NAME <chr>, GEOGRAPHY_CODE <chr>, GEOGRAPHY_TYPE <chr>,
## #   GEOGRAPHY_TYPECODE <chr>, GEOGRAPHY_SORTORDER <chr>, C_TENHUK11 <chr>,
## #   C_TENHUK11_NAME <chr>, C_TENHUK11_CODE <chr>, C_TENHUK11_TYPE <chr>,
## #   C_TENHUK11_TYPECODE <chr>, C_TENHUK11_SORTORDER <chr>,
## #   C_PPBROOMHEW11 <chr>, C_PPBROOMHEW11_NAME <chr>,
## #   C_PPBROOMHEW11_CODE <chr>, C_PPBROOMHEW11_TYPE <chr>,
## #   C_PPBROOMHEW11_TYPECODE <chr>, C_PPBROOMHEW11_SORTORDER <chr>,
## #   C_AHTHUK11 <chr>, C_AHTHUK11_NAME <chr>, C_AHTHUK11_CODE <chr>,
## #   C_AHTHUK11_TYPE <chr>, C_AHTHUK11_TYPECODE <chr>,
## #   C_AHTHUK11_SORTORDER <chr>, MEASURES <chr>, MEASURES_NAME <chr>,
## #   OBS_VALUE <dbl>, OBS_STATUS <chr>, OBS_STATUS_NAME <chr>,
## #   OBS_CONF <chr>, OBS_CONF_NAME <chr>, URN <chr>, RECORD_OFFSET <chr>,
## #   RECORD_COUNT <chr>

We can also query bedroom occupancy per household type in the Manchester, Gorton and Manchester, Withington parliamentary constituencies.

 x <- nomis_get_data(id = "NM_893_1", time = "latest", geography = c("1929380119", "1929380120"))
 
print(x)
## # A tibble: 420 x 40
##    DATE  DATE_NAME DATE_CODE DATE_TYPE DATE_TYPECODE DATE_SORTORDER
##    <chr> <chr>     <chr>     <chr>     <chr>         <chr>         
##  1 2011  2011      2011      date      0             0             
##  2 2011  2011      2011      date      0             0             
##  3 2011  2011      2011      date      0             0             
##  4 2011  2011      2011      date      0             0             
##  5 2011  2011      2011      date      0             0             
##  6 2011  2011      2011      date      0             0             
##  7 2011  2011      2011      date      0             0             
##  8 2011  2011      2011      date      0             0             
##  9 2011  2011      2011      date      0             0             
## 10 2011  2011      2011      date      0             0             
## # ... with 410 more rows, and 34 more variables: GEOGRAPHY <chr>,
## #   GEOGRAPHY_NAME <chr>, GEOGRAPHY_CODE <chr>, GEOGRAPHY_TYPE <chr>,
## #   GEOGRAPHY_TYPECODE <chr>, GEOGRAPHY_SORTORDER <chr>, C_TENHUK11 <chr>,
## #   C_TENHUK11_NAME <chr>, C_TENHUK11_CODE <chr>, C_TENHUK11_TYPE <chr>,
## #   C_TENHUK11_TYPECODE <chr>, C_TENHUK11_SORTORDER <chr>,
## #   C_PPBROOMHEW11 <chr>, C_PPBROOMHEW11_NAME <chr>,
## #   C_PPBROOMHEW11_CODE <chr>, C_PPBROOMHEW11_TYPE <chr>,
## #   C_PPBROOMHEW11_TYPECODE <chr>, C_PPBROOMHEW11_SORTORDER <chr>,
## #   C_AHTHUK11 <chr>, C_AHTHUK11_NAME <chr>, C_AHTHUK11_CODE <chr>,
## #   C_AHTHUK11_TYPE <chr>, C_AHTHUK11_TYPECODE <chr>,
## #   C_AHTHUK11_SORTORDER <chr>, MEASURES <chr>, MEASURES_NAME <chr>,
## #   OBS_VALUE <dbl>, OBS_STATUS <chr>, OBS_STATUS_NAME <chr>,
## #   OBS_CONF <chr>, OBS_CONF_NAME <chr>, URN <chr>, RECORD_OFFSET <chr>,
## #   RECORD_COUNT <chr>