Load Data

Functions that load the data for the map.

Contents

Solomon Geospatial Data - SolomonGeo: A class that cleans the solomon islandas census and geography data - SolomonGeo.read_test: Loads and transforms the test data - SolomonGeo.get_geojson: Returns the geo_df as a geojson datatset

Solomon Geospatial Data

Load the geography and census data ### Geography Data Solomons islands geography data is organised at the levels - adm0 - The country as as whole, Solomon Islands - adm1 - Also referred to as the province e.g. Honiara, Malaita - adm2 - The Consituency e.g. Central Honiara - adm3 - Ward, the smallest geography I am reporting. E.g. Cruz

Census Data

Solomon islands census data has been used from the 2009 and 2019 census. For the respective census: 2009 - We have the total population in for each of the administration regions 2019 - There is only data available down to the province level

Test that the .env variables exist

# Test that the environment variables can be loaded
test(ACCESS_KEY, None, nequals)
test(SECRET_ACCCESS_KEY, None, nequals)
test(REGION_NAME, None, nequals)
test_eq(REGION_NAME, 'ap-southeast-2')

Function for connecting to AWS S3 client

Load and Prepare Elections Data

Load and manipulate the elections data. Below data will be manipulated in SolomonGeo.__clean_elections function.

elec = pd.read_csv(fp + 'Solomon_Elections_Data_April_2024.csv', thousands=',')
#elec['Party'] = 
candParty = elec['Candidate'].str.split("(", expand = True)
elec['candParty'] = elec['Candidate']
elec['Candidate'] = candParty.loc[:, 0]
elec['Party'] = candParty.loc[:, 1].str.strip(")")

# Fix formatting
#elec.Total.astype(int)
elec.sort_values(by = ['Type','Year', 'Geo', 'loc_name', 'Total',], inplace = True, ascending= False)
elec['Order'] = elec.groupby(by = 'loc_name').cumcount()
elec.sort_values(by = ['Type','Year', 'Geo', 'loc_name', 'Total',], inplace = True, ascending= True)
elec.head()
Type Year Geo loc_name Candidate Total Proportion candParty Party Order
4 National Parliament 2024 Constituency Auki-Langalanga Joseph Waleanisia 108 1.7 Joseph Waleanisia (UP) UP 4
3 National Parliament 2024 Constituency Auki-Langalanga Obadiah Koti 257 3.9 Obadiah Koti (PLD) PLD 3
2 National Parliament 2024 Constituency Auki-Langalanga Talauburi Anisi 488 7.5 Talauburi Anisi (KAD) KAD 2
1 National Parliament 2024 Constituency Auki-Langalanga David Filia 1348 20.7 David Filia (OUR) OUR 1
0 National Parliament 2024 Constituency Auki-Langalanga Cooper Wale 4314 66.2 Cooper Wale (SID) SID 0

Make a wide dataset that has the top three finishing candidates per election and electorate.

elec_wide = elec.copy()
elec_wide = elec_wide[elec_wide['Order'] <= 2]
elec_wide['displayValue'] = elec_wide.candParty.map(str) + " - " + elec_wide.Total.map(str) + " (" + elec_wide.Proportion.map(str) + "%)"
elec_wide = pd.pivot(elec_wide, index = ['Type', 'Year', 'Geo', 'loc_name',], columns = ['Order', ], values = ['displayValue']).fillna("")
elec_wide = elec_wide.droplevel(0, axis = 1)
elec_wide.head()
Order 0 1 2
Type Year Geo loc_name
National Parliament 2024 Constituency Auki-Langalanga Cooper Wale (SID) - 4314 (66.2%) David Filia (OUR) - 1348 (20.7%) Talauburi Anisi (KAD) - 488 (7.5%)
Baegu-Asifola Tagini Makario (OUR) - 2391 (37.4%) Celsus Talifilu (U4C) - 2147 (33.6%) Douglas Buga (SIP) - 1009 (15.8%)
Central Guadalcanal Shanel Agovaka (OUR) - 3454 (43.1%) Manengelea Rex (IND) - 1592 (19.8%) Denson Denni (KAD) - 1381 (17.2%)
Central Honiara Darcy Lilo (SIP) - 8223 (63.4%) Alfred Efona (UP) - 3632 (28.0%) Francis Idu (OUR) - 669 (5.2%)
Central Kwara'ae Ricky Fuo'o (UP) - 5208 (47.9%) Jackson Fiulaua (OUR) - 1831 (16.8%) Young Wate (IND) - 1537 (14.1%)

Find winning candidate in each location and election

elec_wide2 = elec.copy().loc[elec.Order == 0, :]
elec_wide2.set_index(['Type', 'Year', 'Geo', 'loc_name',], inplace = True)

elec_wide2.head()
Candidate Total Proportion candParty Party Order
Type Year Geo loc_name
National Parliament 2024 Constituency Auki-Langalanga Cooper Wale 4314 66.2 Cooper Wale (SID) SID 0
Baegu-Asifola Tagini Makario 2391 37.4 Tagini Makario (OUR) OUR 0
Central Guadalcanal Shanel Agovaka 3454 43.1 Shanel Agovaka (OUR) OUR 0
Central Honiara Darcy Lilo 8223 63.4 Darcy Lilo (SIP) SIP 0
Central Kwara'ae Ricky Fuo'o 5208 47.9 Ricky Fuo'o (UP) UP 0

Merge back together the two datasets

elec_wide2 = pd.DataFrame(elec_wide2.Party)
elec_wide2.rename(columns = {"Party": "Winning Party"}, inplace = True)
elec_wide = pd.merge(elec_wide, elec_wide2, how = 'left', left_index = True, right_index = True)
# remove the index
elec_wide.reset_index(inplace = True)
elec.Party.unique()
array(['UP', 'PLD', 'KAD', 'OUR', 'SID', 'IND', 'SIP', 'U4C', 'PAP',
       'PFP', 'DAP', 'NTP', 'GPS'], dtype=object)

Make the Proportion column a decimal to be displayed properly

elec.Proportion = elec.Proportion / 100

SolomonGeo Data Class

This data class manipulates and stores the geospatial and regular data relating to each geography. It includes the necessary data and methods to manipulate that data to build the resulting dash app.

Constructors for SolomonGeo class

These constudctors are used to build the SolomonGeo objects from various inputs


source

SolomonGeo.load_pickle

def load_pickle(
    folder:str='/testData/', # file path of the folder to save in
    aws:bool=True, # Whether to load from github or local
    file_name:str='sol_geo.pickle', # file name of the saved class
):

A constuctor that initialises the object from aws pickle

Test that dataset and geography merge perfectly

# TODO should this method be callable from the class
def test_geo(df:pd.DataFrame, # The dataframe to test
                geo:gpd.GeoDataFrame, # The geographys
                ):
    '''
    Tests that the provided geography and data merge perfectly. I.e. all data
    has a coresponding geography. The primary key to merge id and agg
    '''
    # Use outer join to see if anything is missing
    geo_df = geo.merge(df, left_on=['id', 'agg'], right_on=['core: id', 'core: agg'], how = 'outer', indicator = True)
    test_eq(sum(geo_df['_merge'] != 'both'), 0) # The total of non-joined rows should be 0

Save SolomonGeo

Save as a pickled object


source

SolomonGeo.save_pickle

def save_pickle(
    aws:bool=True, # Whether to save to aws or locally
    folder:str='/testData/', # file path of the folder to save in, only necesasry for local saving
    file_name:str='sol_geo.pickle', # file name of the saved class
):

Save a pickle of the SolomonGeo class in backblaze b2


source

SolomonGeo.save_pickle

def save_pickle(
    aws:bool=True, # Whether to save to aws or locally
    folder:str='/testData/', # file path of the folder to save in, only necesasry for local saving
    file_name:str='sol_geo.pickle', # file name of the saved class
):

Save a pickle of the SolomonGeo class in backblaze b2

Get Geo JSON

A getter method for the geometry portion of the dataset that returns a geoJson formated geography.

It only includes the geography and location name as id


source

SolomonGeo.get_geojson

def get_geojson(
    geo_filter:str=None, # Filters the geojson to the requested aggregation
)->dict: # Geo JSON formatted dataset

A getter method for the SolomonGeo class that returns a Geo JSON formatted dataset

Store JSON

A getter method that returns a dcc.Store object with the data of the SolomonGeo class converted to a dictionary for storing with dash. We use a dictionary instead of geojson as the time to convert dictionary back to geopandas is much faster than geojson. (0.01s vs 1.2s) This makes the map much more snappy and responsive

On storing, we drop the geometry as it won’t store as json. Note that you can get around this by using a geojson, but this increases reload time by a lot.


source

SolomonGeo.get_store

def get_store()->Store: # Geo JSON formatted dataset

A getter method that returns a dcc.Store object with the data of the SolomonGeo class converted to a dictionary for storing with dash.

Get Census Data

Returns the census dataset for a particular aggregation


source

SolomonGeo.get_census

def get_census(
    geo_filter:str=None, # Filters the dataframe to the requested geography
    var:str='Key Statistics', # Selects an upper level
    measure:str=None, # Selects the lower level variable, if var 1 is used, measure must be used.
    loc_filter:[<class 'str'>]=None, # Filters one of more locations
    type_filter:str='Total', # Return either number of proportion
    agg:bool=False, # Whether to return the dataset aggregated for the given selection
)->DataFrame: # Pandas Dataframe containing population data

A getter method for the SolomonGeo class that returns a pandas dataset containg the id variable and the requested census data. This is the minimal data required to display on the map. - Optionally can aggregate the dataset, uses weighted aggregation for proportional data - You should always provide a variable to this function or proportional aggregation won’t work

Get Population Data

Returns the population projection data for a given aggregation


source

SolomonGeo.get_pop

def get_pop(
    years:[<class 'str'>], # Selects the year/years of data to return
    var:str=None, # Selects an upper level variable
    measure:str=None, # Selects the lower level variable, if var 1 is used, measure must be used.
    loc_filter:[<class 'str'>]=None, # Filters one of more locations
    type_filter:str='Total', # Return either number of proportion
    agg:bool=False, # Whether to return the dataset aggregated for the given selection
    agg_location:bool=False, # If true, don't aggregate the population data by location
    agg_ages:bool=False, # If true, don't aggregate the population data by age
    ages:[<class 'str'>]=None, # Filters for one or more Age Brackets, if none returns all
)->DataFrame: # Pandas Dataframe containing population data

A getter method for the SolomonGeo class that returns a pandas dataset containg the id variable and the requested popultion data. This is the minimal data required to display on the map.

Testing

sol_geo = SolomonGeo.read_test()
/tmp/ipykernel_4212/1241796792.py:276: PerformanceWarning: DataFrame is highly fragmented.  This is usually the result of calling `frame.insert` many times, which has poor performance.  Consider joining all columns at once using pd.concat(axis=1) instead. To get a de-fragmented frame, use `newframe = frame.copy()`
  df['pk'] = df['location'] + "_" + df["type"]
test = sol_geo.population.copy()
test['Age']['Age_Bracket']
(core, location)
Central      0-4
Central      0-4
Central      5-9
Central      5-9
Central    10-14
           ...  
Western    70-74
Western    75-79
Western    75-79
Western      80+
Western      80+
Name: Age_Bracket, Length: 5440, dtype: object
tester = sol_geo.get_pop(years = [2010, 2011, 2012], var = "Population", measure='Total').values[:, -1]
tester
# TODO work out how to aggregate all the columns other than age.
array([4156.0, 3988.0, 3289.0, 2521.0, 2118.0, 2102.0, 2022.0, 1940.0,
       1382.0, 1126.0, 821.0, 694.0, 563.0, 428.0, 293.0, 212.0, 199.0,
       4115.0, 4064.0, 3444.0, 2638.0, 2131.0, 2060.0, 2022.0, 1953.0,
       1498.0, 1139.0, 879.0, 695.0, 585.0, 442.0, 306.0, 209.0, 209.0,
       4062.0, 4112.0, 3594.0, 2772.0, 2164.0, 2018.0, 2022.0, 1944.0,
       1623.0, 1150.0, 944.0, 697.0, 606.0, 456.0, 321.0, 210.0, 217.0,
       4339.0, 4133.0, 3418.0, 2721.0, 2275.0, 2072.0, 1916.0, 1841.0,
       1431.0, 1147.0, 835.0, 706.0, 509.0, 483.0, 311.0, 207.0, 136.0,
       4366.0, 4211.0, 3580.0, 2842.0, 2345.0, 2103.0, 1946.0, 1863.0,
       1528.0, 1186.0, 892.0, 715.0, 543.0, 471.0, 344.0, 206.0, 151.0,
       4397.0, 4262.0, 3739.0, 2971.0, 2426.0, 2137.0, 1981.0, 1873.0,
       1629.0, 1224.0, 958.0, 722.0, 584.0, 454.0, 378.0, 206.0, 167.0,
       16116.0, 14274.0, 11675.0, 9959.0, 10071.0, 8908.0, 7662.0, 6667.0,
       4865.0, 3772.0, 2509.0, 2035.0, 1456.0, 1248.0, 878.0, 561.0,
       404.0, 16545.0, 14767.0, 12222.0, 10344.0, 10327.0, 9569.0, 8114.0,
       7060.0, 5352.0, 3998.0, 2795.0, 2095.0, 1582.0, 1254.0, 944.0,
       583.0, 430.0, 17002.0, 15222.0, 12794.0, 10778.0, 10513.0, 10227.0,
       8595.0, 7434.0, 5874.0, 4224.0, 3115.0, 2154.0, 1724.0, 1257.0,
       1012.0, 602.0, 461.0, 8338.0, 7292.0, 6381.0, 7062.0, 9399.0,
       8235.0, 6289.0, 5060.0, 3667.0, 2806.0, 1995.0, 1393.0, 910.0,
       577.0, 312.0, 173.0, 114.0, 8578.0, 7491.0, 6504.0, 6806.0, 9079.0,
       8811.0, 6668.0, 5315.0, 3968.0, 2946.0, 2161.0, 1488.0, 989.0,
       623.0, 348.0, 178.0, 125.0, 8794.0, 7688.0, 6689.0, 6628.0, 8620.0,
       9341.0, 7078.0, 5569.0, 4292.0, 3090.0, 2335.0, 1589.0, 1075.0,
       671.0, 388.0, 188.0, 135.0, 4023.0, 3898.0, 3206.0, 2615.0, 2188.0,
       2053.0, 2039.0, 1893.0, 1339.0, 1137.0, 955.0, 823.0, 634.0, 499.0,
       350.0, 268.0, 228.0, 4029.0, 3965.0, 3364.0, 2712.0, 2261.0,
       2063.0, 2047.0, 1948.0, 1457.0, 1145.0, 995.0, 839.0, 666.0, 511.0,
       367.0, 262.0, 246.0, 4035.0, 4001.0, 3523.0, 2815.0, 2346.0,
       2080.0, 2051.0, 1984.0, 1587.0, 1155.0, 1036.0, 854.0, 700.0,
       522.0, 387.0, 260.0, 260.0, 7400.0, 6377.0, 5042.0, 3807.0, 3671.0,
       3553.0, 3199.0, 2734.0, 1892.0, 1542.0, 1132.0, 958.0, 740.0,
       629.0, 399.0, 295.0, 257.0, 7434.0, 6625.0, 5301.0, 3990.0, 3613.0,
       3587.0, 3263.0, 2831.0, 2050.0, 1566.0, 1212.0, 964.0, 776.0,
       630.0, 433.0, 282.0, 275.0, 7420.0, 6855.0, 5560.0, 4220.0, 3563.0,
       3612.0, 3321.0, 2909.0, 2227.0, 1590.0, 1301.0, 969.0, 816.0,
       627.0, 469.0, 274.0, 288.0, 22242.0, 21688.0, 18652.0, 14923.0,
       12290.0, 10767.0, 9563.0, 8157.0, 6171.0, 5141.0, 4105.0, 3477.0,
       3054.0, 2348.0, 1540.0, 1056.0, 969.0, 21921.0, 21930.0, 19311.0,
       15420.0, 12354.0, 10667.0, 9526.0, 8273.0, 6423.0, 5177.0, 4244.0,
       3487.0, 3078.0, 2429.0, 1613.0, 1039.0, 1014.0, 21554.0, 22035.0,
       19931.0, 15990.0, 12490.0, 10579.0, 9467.0, 8346.0, 6698.0, 5209.0,
       4394.0, 3509.0, 3091.0, 2504.0, 1691.0, 1046.0, 1041.0, 409.0,
       414.0, 448.0, 301.0, 251.0, 169.0, 204.0, 188.0, 134.0, 134.0,
       138.0, 146.0, 111.0, 58.0, 62.0, 62.0, 46.0, 409.0, 417.0, 445.0,
       341.0, 255.0, 190.0, 193.0, 201.0, 144.0, 130.0, 138.0, 145.0,
       120.0, 66.0, 56.0, 60.0, 51.0, 411.0, 421.0, 436.0, 383.0, 258.0,
       215.0, 181.0, 212.0, 156.0, 128.0, 138.0, 142.0, 128.0, 75.0, 51.0,
       60.0, 56.0, 3048.0, 3293.0, 2702.0, 2184.0, 1727.0, 1555.0, 1390.0,
       1416.0, 1153.0, 967.0, 783.0, 687.0, 499.0, 386.0, 320.0, 267.0,
       302.0, 2985.0, 3294.0, 2849.0, 2245.0, 1776.0, 1537.0, 1393.0,
       1389.0, 1211.0, 980.0, 811.0, 694.0, 532.0, 388.0, 319.0, 262.0,
       306.0, 2933.0, 3251.0, 2995.0, 2312.0, 1839.0, 1520.0, 1405.0,
       1348.0, 1269.0, 993.0, 842.0, 697.0, 567.0, 393.0, 320.0, 258.0,
       310.0, 11916.0, 11508.0, 11564.0, 11520.0, 10650.0, 8704.0, 7373.0,
       6083.0, 5447.0, 4995.0, 4729.0, 4054.0, 3076.0, 2080.0, 1468.0,
       959.0, 897.0, 11911.0, 11346.0, 9841.0, 8485.0, 7192.0, 6216.0,
       5623.0, 5244.0, 4673.0, 3734.0, 2723.0, 2070.0, 1621.0, 1250.0,
       906.0, 630.0, 466.0, 11914.0, 11481.0, 10203.0, 8634.0, 7427.0,
       6268.0, 5690.0, 5252.0, 4801.0, 3904.0, 2894.0, 2131.0, 1689.0,
       1278.0, 944.0, 632.0, 508.0], dtype=object)
sol_geo.get_pop(years = [2010, 2011, 2012], var = 'Population', measure = 'Total',  ages = ["0-4"])#, agg = True) loc_filter = ['Honiara', 'Choiseul'],
core Age Population
year Age_Bracket Total
(core, location)
Central 2010 0-4 4156.0
Central 2011 0-4 4115.0
Central 2012 0-4 4062.0
Choiseul 2010 0-4 4339.0
Choiseul 2011 0-4 4366.0
Choiseul 2012 0-4 4397.0
Guadalcanal 2010 0-4 16116.0
Guadalcanal 2011 0-4 16545.0
Guadalcanal 2012 0-4 17002.0
Honiara 2010 0-4 8338.0
Honiara 2011 0-4 8578.0
Honiara 2012 0-4 8794.0
Isabel 2010 0-4 4023.0
Isabel 2011 0-4 4029.0
Isabel 2012 0-4 4035.0
Makira-Ulawa 2010 0-4 7400.0
Makira-Ulawa 2011 0-4 7434.0
Makira-Ulawa 2012 0-4 7420.0
Malaita 2010 0-4 22242.0
Malaita 2011 0-4 21921.0
Malaita 2012 0-4 21554.0
Rennell-Bell 2010 0-4 409.0
Rennell-Bell 2011 0-4 409.0
Rennell-Bell 2012 0-4 411.0
Temotu 2010 0-4 3048.0
Temotu 2011 0-4 2985.0
Temotu 2012 0-4 2933.0
Western 2010 0-4 11916.0
Western 2011 0-4 11911.0
Western 2012 0-4 11914.0
sol_geo.locations
{'Province': array(['Central', 'Choiseul', 'Guadalcanal', 'Honiara', 'Isabel',
        'Makira-Ulawa', 'Malaita', 'Rennell-Bell', 'Temotu', 'Western'],
       dtype=object),
 'Constituency': array(['Auki-Langalanga', 'Baegu-Asifola', 'Central Guadalcanal',
        'Central Honiara', "Central Kwara'ae", 'Central Makira',
        'East AreAre', 'East Choiseul', 'East Guadalcanal', 'East Honiara',
        'East Kwaio', 'East Makira', 'East Malaita', 'East-Central',
        'Fataleka', 'Gao-Bugotu', 'Gizo-Kolombangara',
        'Hograno-Kia-Havulei', 'Lau-Mbaelela', 'Malaita Outer Island',
        'Marigne-Kokota', 'Marovo', 'Nggela', 'North Guadalcanal',
        'North Malaita', 'North New Georgia', 'North Vella la Vella',
        'North West Choiseul', 'North West Guadalcanl', 'North-East',
        'Ranongga-Simbo', 'Rennell-Bellona', 'Savo-Russull Islands',
        'Shortland Islands', 'Small Malaita', 'South Choiseul',
        'South Guadalcanal', 'South New Georgia-Rendova',
        'South Vella la Vella', 'Temotu Pele', 'Temotu Vatu',
        'Temotu-Nende', 'Ulawa-Ugi', 'West Areare', 'West Guadalcanal',
        'West Honiara', 'West Kwaio', "West Kwara'ae", 'West Makira',
        'West New Georgia-Vonavona'], dtype=object),
 'Ward': array(['Aba - Asimeuru', 'Aiaisi', 'Aimela', 'Aola', 'Areare',
        'Arosi East', 'Arosi North', 'Arosi South', 'Arosi West', 'Asimae',
        'Auki', 'Avuavu', 'Babatana', 'Bangara', 'Batava', 'Bauro Central',
        'Bauro East', 'Bauro West', 'Bilua', 'Birao', 'Buala',
        'Buini Tusu', 'Buma', 'Central Ranongga', 'Cruz', 'Dovele',
        'Duff Islands', 'Duidui', 'East Baegu', 'East Gela',
        'East Ghaobata', 'East Ghongau', 'East Tasimboko',
        'East Te Nggano', 'Fauabu', 'Faumamanu - Kwai', 'Fenualoa',
        'Foondo - Gwaiau', 'Fouenda', 'Gizo', 'Graciosa Bay', 'Gulalafou',
        'Haununu', 'Havulei', 'Hovukoilo', 'Inner Shortlands', 'Iringgila',
        'Japuana', 'Kaloka', 'Kangava', 'Katupika', 'Kerepangara', 'Kia',
        'Kirugela', 'Kmaga', 'Kokota', "Kola'A", 'Kolokarako',
        'Kolombaghea', 'Kolomola', 'Kolotubi', 'Koviloko', 'Kukum',
        'Kusaghe', 'Kwaimela - Radefasu', 'Kwarekwareo', 'Langalanga',
        'Lipe-Temua', 'Longgu', 'Lovukol', 'Luaniua', 'Lughu',
        'Luva Station', 'Malango', "Malu'U", 'Mandalua - Folotana',
        'Manuopo', 'Mareho', 'Matakwalao', 'Matangi', 'Mataniko',
        'Mbanika', 'Mbumburu', 'Moli', 'Mugihenua', 'Munda', 'Nafinua',
        'Naha', 'Nanggu-Lord Howe', 'Nea-Noole', 'Nenumpo', 'Neo',
        'Nevenema', 'Nggatokae', 'Nggosi', 'Nipua-Nopoli', 'Nono', 'Noro',
        'North East Gela', 'North East Santa Cruz', 'North Kolombangara',
        'North Ranongga', 'North Rendova', 'North Savo', 'North Ulawa',
        'North Vangunu', 'North West Gela', 'Nusa Roviana',
        'Outer Shortlands', 'Panatina', 'Paripao', 'Pavuvu', 'Pelau',
        'Polo', 'Polynesian Outer Islands', 'Raroisuu', 'Rawo',
        'Rove - Lengakiki', 'Roviana Lagoon', "Sa'Aiho", 'Saghalu',
        'Samasodu', 'Sandfly-Buenavista', 'Santa Ana', 'Santa Catalina',
        'Savulei', 'Senga', 'Siesie', 'Sigana', 'Sikaiana', 'Simbo',
        'South East Gela', 'South Kolombangara', 'South Ranongga',
        'South Rendova', 'South Savo', 'South Ulawa', 'South West Gela',
        'Star Harbour North', 'Star Harbour South', 'Sububenu- Burianias',
        'Sulufou - Kwarande', 'Susubona', 'Susuka', 'Tai', 'Takwa',
        'Talise', 'Tandai', 'Tangarare', 'Tatamba', 'Tavula', 'Tepakaza',
        'Tetau Nangoto', 'Tetekanji', 'Tikopia', 'Tirotonga', 'Tulagi',
        'Ugi And Pio', 'Utupua', 'Valasi', 'Vanikoro', 'Vasiduki',
        'Vatukulau', 'Vavaea', 'Viviru', 'Vonavona', 'Vonunu', 'Vuhokesa',
        'Vulolo', 'Vura', 'Waghina', 'Wainoni East', 'Wainoni West',
        'Wanderer Bay', 'Waneagu - Taelanasin', 'Weagu Silana Sina',
        'Weather Coast', 'West Baegu \xa0- Fatale', 'West Gela',
        'West Ghaobata', 'West Ghongau', 'West Te Nggano', 'West Ulawa'],
       dtype=object)}

Save and Load

Test that the newly created solomon geo object can be saved to aws

sol_geo.save_pickle()

Test that we can connect to the aws s3 bucket

# Create the S3 object
obj = s3_client().get_object(
    Bucket = 'hobby-data',
    Key = 'test.txt', 
)

# Read in the pickle
try:
    data = obj['Body'].read()
except:
    raise ValueError("Issue dowloading test file from AWS.")
SolomonGeo.load_pickle('/testData/', aws=True)
<__main__.SolomonGeo>

Test that gen_stored creates a copy correctly from a json serialised dataframe

stored_geo = sol_geo.get_store()
# TODO I need to potentially created a function to check if two objects are the same
pd.DataFrame(stored_geo.data['data']['elec'])
Type Year Geo loc_name Candidate Total Proportion candParty Party Order
0 National Parliament 2024 Constituency Auki-Langalanga Joseph Waleanisia 108 0.017 Joseph Waleanisia (UP) UP 4
1 National Parliament 2024 Constituency Auki-Langalanga Obadiah Koti 257 0.039 Obadiah Koti (PLD) PLD 3
2 National Parliament 2024 Constituency Auki-Langalanga Talauburi Anisi 488 0.075 Talauburi Anisi (KAD) KAD 2
3 National Parliament 2024 Constituency Auki-Langalanga David Filia 1348 0.207 David Filia (OUR) OUR 1
4 National Parliament 2024 Constituency Auki-Langalanga Cooper Wale 4314 0.662 Cooper Wale (SID) SID 0
... ... ... ... ... ... ... ... ... ... ...
329 National Parliament 2024 Constituency West New Georgia-Vonavona Usa Kama 11 0.001 Usa Kama (IND) IND 4
330 National Parliament 2024 Constituency West New Georgia-Vonavona Ben Anikwai 45 0.006 Ben Anikwai (IND) IND 3
331 National Parliament 2024 Constituency West New Georgia-Vonavona Trevor Veo 1514 0.190 Trevor Veo (OUR) OUR 2
332 National Parliament 2024 Constituency West New Georgia-Vonavona V Tausinga 2524 0.317 V Tausinga (UP) UP 1
333 National Parliament 2024 Constituency West New Georgia-Vonavona George Temahua 3869 0.486 George Temahua (SID) SID 0

334 rows × 10 columns

restored_geo = SolomonGeo.gen_stored(stored_geo.data)
restored_geo.get_census(geo_filter = 'Constituency', loc_filter=['Nggela', 'East AreAre'])
Total Households Total population Male population Female population
(core, location)
East AreAre 1105 7099 3567.0 3532.0
Nggela 3315 16811 8500.0 8311.0
# TODO create a test to check that two object are the same. Probably need to write defualt behaviour into
# object. Maybe use fastcore test_eq
#test_eq(stored_geo.geo_df, sol_geo.geo_df)

Test filtering of down to multiple locations

#test_eq(sol_geo.get_census(geo_filter = 'constituency')['total_pop'].sum(), const_df['total_pop'].sum())
sol_geo.get_census(geo_filter = 'Constituency', loc_filter=['Nggela', 'East AreAre'])
Total Households Total population Male population Female population
(core, location)
East AreAre 1105 7099 3567.0 3532.0
Nggela 3315 16811 8500.0 8311.0

Test Proportions

#test_eq(sol_geo.get_census(geo_filter = 'constituency')['total_pop'].sum(), const_df['total_pop'].sum())
pd.DataFrame(sol_geo.get_census(geo_filter = 'Constituency', loc_filter=['Nggela', 'East AreAre'], agg = True, type_filter='Proportion'))
0
Total Households 0.084609
Total population 0.457695
Male population 0.230992
Female population 0.226704
# TODO work out how to check save and load are the same as a test
# Might need to wreite and equality function
restored_geo.population
core Age Population
id agg type year Age_Bracket Numerical_Bracket Males Females Total
(core, location)
Central 4 Province Total 2010 0-4 0 2119.000000 2037.000000 4156.000000
Central 4 Province Proportion 2010 0-4 0 0.007116 0.007210 0.007162
Central 4 Province Total 2010 5-9 5 2074.000000 1914.000000 3988.000000
Central 4 Province Proportion 2010 5-9 5 0.006965 0.006775 0.006872
Central 4 Province Total 2010 10-14 10 1732.000000 1556.000000 3289.000000
... ... ... ... ... ... ... ... ... ...
Western 2 Province Proportion 2025 70-74 70 0.001877 0.001965 0.001920
Western 2 Province Total 2025 75-79 75 466.000000 493.000000 959.000000
Western 2 Province Proportion 2025 75-79 75 0.001197 0.001315 0.001255
Western 2 Province Total 2025 80+ 80 448.000000 449.000000 897.000000
Western 2 Province Proportion 2025 80+ 80 0.001150 0.001197 0.001173

5440 rows × 9 columns

test = sol_geo.population
test.loc[:, ('core', 'location')] = test.index
test
core Age Population core
id agg type year Age_Bracket Numerical_Bracket Males Females Total location
(core, location)
Central 4 Province Total 2010 0-4 0 2119.000000 2037.000000 4156.000000 Central
Central 4 Province Proportion 2010 0-4 0 0.007116 0.007210 0.007162 Central
Central 4 Province Total 2010 5-9 5 2074.000000 1914.000000 3988.000000 Central
Central 4 Province Proportion 2010 5-9 5 0.006965 0.006775 0.006872 Central
Central 4 Province Total 2010 10-14 10 1732.000000 1556.000000 3289.000000 Central
... ... ... ... ... ... ... ... ... ... ...
Western 2 Province Proportion 2025 70-74 70 0.001877 0.001965 0.001920 Western
Western 2 Province Total 2025 75-79 75 466.000000 493.000000 959.000000 Western
Western 2 Province Proportion 2025 75-79 75 0.001197 0.001315 0.001255 Western
Western 2 Province Total 2025 80+ 80 448.000000 449.000000 897.000000 Western
Western 2 Province Proportion 2025 80+ 80 0.001150 0.001197 0.001173 Western

5440 rows × 10 columns

# TODO should do a count check for the geojson similar to this
sol_geo.census[sol_geo.census['core']['agg'] == 'Constituency'].count()
core                    id                         100
                        location                   100
                        agg                        100
                        type                       100
Key Statistics          Total Households           100
                                                  ... 
Disability walking      Some Difficulty            100
                        Cannot perform the task    100
Disability remembering  No Difficulty at all       100
                        Some Difficulty            100
                        Cannot perform the task    100
Length: 339, dtype: int64

TODO check that all proportions are less than 0

# TODO filter and test equality?
test = sol_geo.get_geojson(geo_filter = 'Ward')
sol_geo.geo
geometry id agg year location
location
Aba - Asimeuru MULTIPOLYGON (((161.54336 -9.54751, 161.54367 ... 722 Ward 2009 Aba - Asimeuru
Aiaisi MULTIPOLYGON (((161.26900 -9.21229, 161.27144 ... 719 Ward 2009 Aiaisi
Aimela MULTIPOLYGON (((160.68653 -8.71158, 160.67020 ... 702 Ward 2009 Aimela
Aola MULTIPOLYGON (((160.50139 -9.52996, 160.50196 ... 616 Ward 2009 Aola
Areare MULTIPOLYGON (((161.35878 -9.46664, 161.35859 ... 720 Ward 2009 Areare
... ... ... ... ... ...
West Makira MULTIPOLYGON (((161.50441 -10.51180, 161.50559... 844 Constituency 2009 West Makira
West New Georgia-Vonavona MULTIPOLYGON (((157.27598 -8.35143, 157.26991 ... 209 Constituency 2009 West New Georgia-Vonavona
West Te Nggano MULTIPOLYGON (((160.35206 -11.66366, 160.30721... 502 Ward 2009 West Te Nggano
West Ulawa MULTIPOLYGON (((161.97730 -9.77521, 161.94899 ... 803 Ward 2009 West Ulawa
Western MULTIPOLYGON (((158.25148 -8.78861, 158.24885 ... 2 Province 2009 Western

243 rows × 5 columns

The sum after merging and filtering should equal the sum from the raw dataset.

test_eq(sol_geo.get_census(geo_filter = 'Constituency',var = 'Key Statistics', measure = 'Total Households', agg = True)[0], 
        const_df.loc[const_df['type'] == 'Total'].loc[const_df['agg'] == 'Constituency']['Key Statistics: Total Households'].sum())
/tmp/ipykernel_4212/2628698497.py:1: FutureWarning: Series.__getitem__ treating keys as positions is deprecated. In a future version, integer keys will always be treated as labels (consistent with DataFrame behavior). To access a value by position, use `ser.iloc[pos]`
  test_eq(sol_geo.get_census(geo_filter = 'Constituency',var = 'Key Statistics', measure = 'Total Households', agg = True)[0],
sol_geo.get_census(geo_filter = 'Constituency', var='Key Statistics', measure = 'Total Households', agg = True)
Total Households    91251
dtype: int64
sol_geo.get_census(geo_filter = 'Constituency', var='Key Statistics', measure = 'Total Households', loc_filter=['West Guadalcanal']).values[0]
array([1671])
sol_geo.census['core'].filter(items = ['id'])
id
pk
Aba - Asimeuru_Proportion 722
Aba - Asimeuru_Total 722
Aiaisi_Total 719
Aiaisi_Proportion 719
Aimela_Proportion 702
... ...
West Te Nggano_Proportion 502
West Ulawa_Proportion 803
West Ulawa_Total 803
Western_Total 2
Western_Proportion 2

486 rows × 1 columns

# TODO -test that they all sum to one
test_agg = sol_geo.get_census(geo_filter = 'Constituency', type_filter='Proportion', var = "Main source of household drinking water", 
                              agg = True)
test_eq(test_agg.sum(), 1.0)
test_agg
metered SIWA drinking water       0.091681
communal standpipe                0.351141
private water tank                0.124711
communal water tank               0.106191
well - protected                  0.020811
well - unprotected                0.017819
river or stream                   0.245433
bottled water                     0.006115
other source of drinking water    0.036098
dtype: float64
sol_geo.get_census(geo_filter = 'Constituency', type_filter='Proportion', var = "Key Statistics", 
                              agg = True)
Total Households     1.000
Total population     1.000
Male population      0.513
Female population    0.487
dtype: float64
test_agg.sum()
1.0
sol_geo.ages
['0-4',
 '10-14',
 '15-19',
 '20-24',
 '25-29',
 '30-34',
 '35-39',
 '40-44',
 '45-49',
 '5-9',
 '50-54',
 '55-59',
 '60-64',
 '65-69',
 '70-74',
 '75-79',
 '80+']
sol_geo.pop_years
[2010,
 2011,
 2012,
 2013,
 2014,
 2015,
 2016,
 2017,
 2018,
 2019,
 2020,
 2021,
 2022,
 2023,
 2024,
 2025]

Testing of Population Getter

sol_geo.get_pop(years = [2024],var='Population',
    measure='Total',
    loc_filter=None,
    type_filter='Total',
    agg=False,
    ages=['0-4'] 
    )
core Age Population
year Age_Bracket Total
(core, location)
Central 2024 0-4 3641.0
Choiseul 2024 0-4 4973.0
Guadalcanal 2024 0-4 22087.0
Honiara 2024 0-4 8535.0
Isabel 2024 0-4 4203.0
Makira-Ulawa 2024 0-4 7242.0
Malaita 2024 0-4 19039.0
Rennell-Bell 2024 0-4 581.0
Temotu 2024 0-4 2697.0
Western 2024 0-4 11834.0
test = sol_geo.get_pop(years = [2023, 2024],var='Population',
    measure='Total',
    loc_filter=None,
    type_filter='Total',
    agg=False,
    ages=['0-4'] 
    )
test = sol_geo.get_pop(years = sol_geo.pop_years,var='Population',
    measure='Total',
    type_filter='Total',
    agg=True,
    agg_location = True,
    ages=sol_geo.ages
    )

Check that sum of proportions rounded is 100

prop = sol_geo.get_pop(years = [sol_geo.pop_years[-1]],
                var='Population',
                measure='Total',
                type_filter='Proportion',
                agg=True,
                ages=sol_geo.ages,
    ).values[0, 0]
test_eq(1, int(prop))
sol_geo.get_pop(years = [2024],
                var='Population',
                measure='Total',
                type_filter='Total',
                agg=True,
                ages=sol_geo.ages,
                agg_location = True,
    )
Population
Total
(core, year) (core, location)
2024 Central 34359.0
Choiseul 40226.0
Guadalcanal 178238.0
Honiara 97801.0
Isabel 38134.0
Makira-Ulawa 59705.0
Malaita 165610.0
Rennell-Bell 4702.0
Temotu 26182.0
Western 105367.0
sol_geo.locations['Province']
array(['Central', 'Choiseul', 'Guadalcanal', 'Honiara', 'Isabel',
       'Makira-Ulawa', 'Malaita', 'Rennell-Bell', 'Temotu', 'Western'],
      dtype=object)
sol_geo.census
core Key Statistics Main source of household drinking water ... Disability seeing Disability hearing Disability walking Disability remembering
id location agg type Total Households metered SIWA drinking water communal standpipe private water tank communal water tank well - protected ... Cannot perform the task No Difficulty at all Some Difficulty Cannot perform the task No Difficulty at all Some Difficulty Cannot perform the task No Difficulty at all Some Difficulty Cannot perform the task
pk
Aba - Asimeuru_Proportion 722 Aba - Asimeuru Ward Proportion 1 0.0000 0.5378 0.0245 0.0085 0.0011 ... 0.003849 0.940843 0.054295 0.004862 0.880065 0.105754 0.014182 0.906807 0.081645 0.011548
Aba - Asimeuru_Total 722 Aba - Asimeuru Ward Total 939 0.0000 505.0000 23.0000 8.0000 1.0000 ... 19.000000 4644.000000 268.000000 24.000000 4344.000000 522.000000 70.000000 4476.000000 403.000000 57.000000
Aiaisi_Total 719 Aiaisi Ward Total 561 1.0000 403.0000 9.0000 3.0000 1.0000 ... 9.000000 3415.000000 149.000000 10.000000 3316.000000 239.000000 19.000000 3349.000000 205.000000 20.000000
Aiaisi_Proportion 719 Aiaisi Ward Proportion 1 0.0018 0.7184 0.0160 0.0053 0.0018 ... 0.002518 0.955512 0.041690 0.002798 0.927812 0.066872 0.005316 0.937045 0.057359 0.005596
Aimela_Proportion 702 Aimela Ward Proportion 1 0.0164 0.2937 0.0762 0.1921 0.1360 ... 0.001571 0.959288 0.038749 0.001964 0.954444 0.039796 0.005760 0.951172 0.043330 0.005498
... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ...
West Te Nggano_Proportion 502 West Te Nggano Ward Proportion 1 0.0000 0.0000 0.3704 0.4568 0.0000 ... 0.000000 0.978836 0.018519 0.002646 0.968254 0.031746 0.000000 0.986772 0.013228 0.000000
West Ulawa_Proportion 803 West Ulawa Ward Proportion 1 0.0099 0.7673 0.0050 0.0050 0.0000 ... 0.000000 0.929070 0.069767 0.001163 0.886047 0.113953 0.000000 0.920930 0.077907 0.001163
West Ulawa_Total 803 West Ulawa Ward Total 202 2.0000 155.0000 1.0000 1.0000 0.0000 ... 0.000000 799.000000 60.000000 1.000000 762.000000 98.000000 0.000000 792.000000 67.000000 1.000000
Western_Total 2 Western Province Total 13762 37.0000 4151.0000 4596.0000 2426.0000 46.0000 ... 159.000000 72641.000000 3788.000000 220.000000 71496.000000 4644.000000 509.000000 69162.000000 6901.000000 586.000000
Western_Proportion 2 Western Province Proportion 1 0.0027 0.3016 0.3340 0.1763 0.0033 ... 0.002074 0.947710 0.049420 0.002870 0.932771 0.060588 0.006641 0.902321 0.090034 0.007645

486 rows × 339 columns

Check Some of the datasets

necessary??

sol_geo.census
core Key Statistics Main source of household drinking water ... Disability seeing Disability hearing Disability walking Disability remembering
id location agg type Total Households metered SIWA drinking water communal standpipe private water tank communal water tank well - protected ... Cannot perform the task No Difficulty at all Some Difficulty Cannot perform the task No Difficulty at all Some Difficulty Cannot perform the task No Difficulty at all Some Difficulty Cannot perform the task
pk
Aba - Asimeuru_Proportion 722 Aba - Asimeuru Ward Proportion 1 0.0000 0.5378 0.0245 0.0085 0.0011 ... 0.003849 0.940843 0.054295 0.004862 0.880065 0.105754 0.014182 0.906807 0.081645 0.011548
Aba - Asimeuru_Total 722 Aba - Asimeuru Ward Total 939 0.0000 505.0000 23.0000 8.0000 1.0000 ... 19.000000 4644.000000 268.000000 24.000000 4344.000000 522.000000 70.000000 4476.000000 403.000000 57.000000
Aiaisi_Total 719 Aiaisi Ward Total 561 1.0000 403.0000 9.0000 3.0000 1.0000 ... 9.000000 3415.000000 149.000000 10.000000 3316.000000 239.000000 19.000000 3349.000000 205.000000 20.000000
Aiaisi_Proportion 719 Aiaisi Ward Proportion 1 0.0018 0.7184 0.0160 0.0053 0.0018 ... 0.002518 0.955512 0.041690 0.002798 0.927812 0.066872 0.005316 0.937045 0.057359 0.005596
Aimela_Proportion 702 Aimela Ward Proportion 1 0.0164 0.2937 0.0762 0.1921 0.1360 ... 0.001571 0.959288 0.038749 0.001964 0.954444 0.039796 0.005760 0.951172 0.043330 0.005498
... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ...
West Te Nggano_Proportion 502 West Te Nggano Ward Proportion 1 0.0000 0.0000 0.3704 0.4568 0.0000 ... 0.000000 0.978836 0.018519 0.002646 0.968254 0.031746 0.000000 0.986772 0.013228 0.000000
West Ulawa_Proportion 803 West Ulawa Ward Proportion 1 0.0099 0.7673 0.0050 0.0050 0.0000 ... 0.000000 0.929070 0.069767 0.001163 0.886047 0.113953 0.000000 0.920930 0.077907 0.001163
West Ulawa_Total 803 West Ulawa Ward Total 202 2.0000 155.0000 1.0000 1.0000 0.0000 ... 0.000000 799.000000 60.000000 1.000000 762.000000 98.000000 0.000000 792.000000 67.000000 1.000000
Western_Total 2 Western Province Total 13762 37.0000 4151.0000 4596.0000 2426.0000 46.0000 ... 159.000000 72641.000000 3788.000000 220.000000 71496.000000 4644.000000 509.000000 69162.000000 6901.000000 586.000000
Western_Proportion 2 Western Province Proportion 1 0.0027 0.3016 0.3340 0.1763 0.0033 ... 0.002074 0.947710 0.049420 0.002870 0.932771 0.060588 0.006641 0.902321 0.090034 0.007645

486 rows × 339 columns

sol_geo.geo
geometry id agg year location
location
Aba - Asimeuru MULTIPOLYGON (((161.54336 -9.54751, 161.54367 ... 722 Ward 2009 Aba - Asimeuru
Aiaisi MULTIPOLYGON (((161.26900 -9.21229, 161.27144 ... 719 Ward 2009 Aiaisi
Aimela MULTIPOLYGON (((160.68653 -8.71158, 160.67020 ... 702 Ward 2009 Aimela
Aola MULTIPOLYGON (((160.50139 -9.52996, 160.50196 ... 616 Ward 2009 Aola
Areare MULTIPOLYGON (((161.35878 -9.46664, 161.35859 ... 720 Ward 2009 Areare
... ... ... ... ... ...
West Makira MULTIPOLYGON (((161.50441 -10.51180, 161.50559... 844 Constituency 2009 West Makira
West New Georgia-Vonavona MULTIPOLYGON (((157.27598 -8.35143, 157.26991 ... 209 Constituency 2009 West New Georgia-Vonavona
West Te Nggano MULTIPOLYGON (((160.35206 -11.66366, 160.30721... 502 Ward 2009 West Te Nggano
West Ulawa MULTIPOLYGON (((161.97730 -9.77521, 161.94899 ... 803 Ward 2009 West Ulawa
Western MULTIPOLYGON (((158.25148 -8.78861, 158.24885 ... 2 Province 2009 Western

243 rows × 5 columns

# TODO write testing that checks that different methods create equilvalent objects.
# TODO - need to write an equality object
test.loc[2024]
Population
Total
(core, location)
Central 34359.0
Choiseul 40226.0
Guadalcanal 178238.0
Honiara 97801.0
Isabel 38134.0
Makira-Ulawa 59705.0
Malaita 165610.0
Rennell-Bell 4702.0
Temotu 26182.0
Western 105367.0
len(sol_geo.pop_years)
16

Test that the election data Exists

sol_geo.elec.head()
Type Year Geo loc_name Candidate Total Proportion candParty Party Order
4 National Parliament 2024 Constituency Auki-Langalanga Joseph Waleanisia 108 0.017 Joseph Waleanisia (UP) UP 4
3 National Parliament 2024 Constituency Auki-Langalanga Obadiah Koti 257 0.039 Obadiah Koti (PLD) PLD 3
2 National Parliament 2024 Constituency Auki-Langalanga Talauburi Anisi 488 0.075 Talauburi Anisi (KAD) KAD 2
1 National Parliament 2024 Constituency Auki-Langalanga David Filia 1348 0.207 David Filia (OUR) OUR 1
0 National Parliament 2024 Constituency Auki-Langalanga Cooper Wale 4314 0.662 Cooper Wale (SID) SID 0
sol_geo.elec_wide.head()
Type Year Geo loc_name 0 1 2 Winning Party
0 National Parliament 2024 Constituency Auki-Langalanga Cooper Wale (SID) - 4314 (66.2%) David Filia (OUR) - 1348 (20.7%) Talauburi Anisi (KAD) - 488 (7.5%) SID
1 National Parliament 2024 Constituency Baegu-Asifola Tagini Makario (OUR) - 2391 (37.4%) Celsus Talifilu (U4C) - 2147 (33.6%) Douglas Buga (SIP) - 1009 (15.8%) OUR
2 National Parliament 2024 Constituency Central Guadalcanal Shanel Agovaka (OUR) - 3454 (43.1%) Manengelea Rex (IND) - 1592 (19.8%) Denson Denni (KAD) - 1381 (17.2%) OUR
3 National Parliament 2024 Constituency Central Honiara Darcy Lilo (SIP) - 8223 (63.4%) Alfred Efona (UP) - 3632 (28.0%) Francis Idu (OUR) - 669 (5.2%) SIP
4 National Parliament 2024 Constituency Central Kwara'ae Ricky Fuo'o (UP) - 5208 (47.9%) Jackson Fiulaua (OUR) - 1831 (16.8%) Young Wate (IND) - 1537 (14.1%) UP