Reproducible workflow using Python
This document provides an example visualisation of the iNaturalist observations from the Gayini wetlands for the ARDC Gayini Trusted Environmental Data and Information Supply Chain project.
The Nari Nari Tribal Council manages and is actively restoring 80,000 ha of the extensive Gayini wetlands on the Murrumbidgee River. With their consortium partners, they are managing environmental flows, feral animals, cultural burning and grazing of livestock. The area is a key breeding area for waterbird rookeries, including the three ibis species, spoonbills, cormorants, herons and Australian pelicans. It also has extensive areas of lignum, river red gum and blackbox as well as terrestrial ecosystems. Nari Nari are supported by three other consortium partners, The Nature Conservancy, Murray Darling Wetlands Working Group and UNSW’s Centre for Ecosystem Science.
Check out the the project page at iNaturalist and the Gayini TEDISC project at ARDC.
This document brings together data from online resources and is meant to be completely reproducible.
I will show here how to download information from a region in iNaturalist, query the observations in that region and visualise the data in three alternative ways: taxonomically, spatially, and temporally.
For this document I am using the get_observations function and the ICONIC_TAXA constants from PyiNaturalist for query and download of the data, Altair and Folium for data visualisation, and some functions from GeoPandas and pandas for convenience in reading data as a data frame, as well as selected functions from the urllib, owslib, json and datetime modules.
from pyinaturalist import get_observations, ICONIC_TAXA
import altair as alt
import folium
import pandas as pd
import geopandas as gpd
from datetime import datetime
from owslib.wms import WebMapService
import urllib.parse, urllib.request, json
from PIL import Image
from io import BytesIO For this workflow we will load observations records and spatial data from iNaturalist and map layers from New South Wales Spatial Services and the Central Resource for Sharing and Enabling Environmental Data in NSW (SEED NSW).
The NSW Base Map layer is available from NSW Spatial Services as a WMTS layer.
NSW_basemap_url = "http://maps.six.nsw.gov.au/arcgis/rest/services/public/NSW_Base_Map/MapServer/WMTS?"
nsw_base_layer = 'public_NSW_Base_Map'
params = {
'Service': 'WMTS',
'Request': 'GetTile',
'Version': '1.0.0',
'Style': 'default',
'tilematrixset': 'default028mm',
'Format': 'image/png',
'layer': nsw_base_layer,
'TileMatrix': '{z}',
'TileRow': '{y}',
'TileCol': '{x}'
}
NSW_basemap_params = urllib.parse.urlencode(params, safe='{}')
NSW_basemap=NSW_basemap_url + NSW_basemap_params
nsw_base_attrib = u" © State of New South Wales, Department of Customer Service, Spatial Services"The spatial data for the Vegetation Formations and Classes of NSW Version 3.03 (Keith and Simpson 2012; updated in 2017) is available from the SEED portal. There are several access options documented in the metadata of the dataset, I use here the Web Map Service.
Vegetation Formations and Classes of NSW (version 3.03 - 200m Raster) - David A. Keith and Christopher C. Simpson. VIS_ID 3848. Updated in 2017 as version 3.1. Available from SEED data portal
md_id = '31986103-db62-4994-9702-054949281f56'
md_url = 'https://datasets.seed.nsw.gov.au/api/3/action/package_show?id=' + md_id
with urllib.request.urlopen(md_url) as url:
metadata = json.load(url)
vegmap_url = metadata['result']['resources'][2]['url']
vegmap_wms = WebMapService(vegmap_url, version="1.3.0")
vegmap_layer = "0"
vegmap_attrib = 'some additional attribution'
wms = vegmap_wms.contents[vegmap_layer]
legendurl = wms.styles['default']['legend']
with urllib.request.urlopen(legendurl) as url:
vegmap_legend = Image.open(BytesIO(url.read()))The pyinaturalist library in Python provides convenient access to the iNaturalist API. We need the place_id that matches the area of interest to query the API with the function get_observations.
PLACE_ID = 209778
PLACE_NAME = 'Gayini wetlands'
observations = get_observations(place_id=PLACE_ID,
per_page=200)iNaturalist also provide access to the spatial information of places that have been contributed by the community. Here we construct a url to access the Gayini wetlands polygon in GeoJSON format.
gayini_geojson = f'https://www.inaturalist.org/places/geometry/{PLACE_ID}.geojson'
#gayini_polygon = gpd.read_file(path)The following snippet of code goes through the list of iNat’s observations (downloaded as a json object or, in this case, a python dictionary), and filters the research quality grade observations to extract records of species names, iconic taxon, date of the observation and the preferred common name, if present. We then summarise the observations grouped by species and count the total number of records per species.
records = list()
for obs in observations['results']:
if obs['quality_grade'] == 'research':
if obs['taxon'] is not None:
record = {
'rank': obs['taxon']['rank'],
'species_name': obs['taxon']['name'],
'iconic_taxon': obs['taxon']['iconic_taxon_name'],
'observed_on': datetime.date(obs['observed_on']),
}
if 'preferred_common_name' in obs['taxon'].keys():
record['common_name']= obs['taxon']['preferred_common_name']
records.append(record)
df = pd.DataFrame(records)
colnames=['iconic_taxon','species_name','common_name','rank',]
df.groupby(colnames)['species_name'].agg([ 'count'])| count | ||||
|---|---|---|---|---|
| iconic_taxon | species_name | common_name | rank | |
| Actinopterygii | Cyprinus carpio | European Carp | species | 2 |
| Gambusia holbrooki | Eastern Mosquitofish | species | 1 | |
| Amphibia | Crinia parinsignifera | Beeping Froglet | species | 1 |
| Litoria peronii | Peron's Tree Frog | species | 2 | |
| Ranoidea raniformis | Southern Bell Frog | species | 4 | |
| Arachnida | Trichonephila edulis | Australian Golden Orbweaver | species | 1 |
| Aves | Anas gracilis | Grey Teal | species | 1 |
| Dromaius novaehollandiae | Emu | species | 2 | |
| Melopsittacus undulatus | Budgerigar | species | 1 | |
| Platalea regia | Royal Spoonbill | species | 1 | |
| Tachybaptus novaehollandiae | Australasian Grebe | species | 1 | |
| Insecta | Ecphantus quadrilobus | Crested Tooth-Grinder | species | 1 |
| Simosyrphus grandicornis | Yellow-shouldered Stout Hover Fly | species | 1 | |
| Mollusca | Succinea australis | Southern Ambersnail | species | 1 |
| Plantae | Atriplex lindleyi | Lindley's Saltbush | species | 1 |
| Cirsium vulgare | Bull Thistle | species | 1 | |
| Limonium lobatum | winged sea-lavender | species | 1 | |
| Paspalum distichum | knot grass | species | 1 | |
| Persicaria prostrata | Creeping Knotweed | species | 1 | |
| Phyla nodiflora | turkey tangle frogfruit | species | 1 | |
| Ptilotus nobilis | Yellow Tails | species | 1 | |
| Sclerolaena brachyptera | Short-wing Copperburr | species | 1 | |
| Sclerolaena muricata | Black Roly-poly | species | 1 | |
| Trifolium resupinatum | Reversed clover | species | 1 | |
| Reptilia | Chelodina longicollis | Eastern Snake-necked Turtle | species | 1 |
| Delma inornata | Olive Delma | species | 1 | |
| Hemiaspis damelii | Grey Snake | species | 3 | |
| Morelia spilota metcalfei | Inland Carpet Python | subspecies | 1 | |
| Notechis scutatus | Tiger Snake | species | 1 | |
| Pseudechis porphyriacus | Red-bellied Black Snake | species | 3 | |
| Pseudonaja textilis | Eastern Brown Snake | species | 1 | |
| Suta suta | Curl Snake | species | 2 | |
| Underwoodisaurus milii | Thick-tailed Barking Gecko | species | 2 | |
| Varanus varius | Lace Monitor | species | 1 |
We can further group the species and observation by groups of iconic taxa.
iconic_df = df.groupby('iconic_taxon')['species_name'].agg([ 'count', 'nunique']).reset_index()
TAXON_IMAGE_URL = 'https://raw.githubusercontent.com/inaturalist/inaturalist/main/app/assets/images/iconic_taxa/{taxon}-75px.png'
iconic_df['img']=iconic_df.iconic_taxon.apply(lambda x: TAXON_IMAGE_URL.format(taxon=x.lower()))
bar1 = alt.Chart(
iconic_df
).mark_bar(
color='grey', opacity=0.15
).encode(
x=alt.X('iconic_taxon:N', sort='-y'),
y='count:Q'
)
bar2 = alt.Chart(
iconic_df
).mark_bar(
color='blue', opacity=0.15
).encode(
x=alt.X('iconic_taxon:N', sort='-y'),
y='nunique:Q'
)
img = alt.Chart(
iconic_df,
title=f'Research grade observations in {PLACE_NAME} by iconic taxon',
width=500,
height=400,
).mark_image(
baseline='top'
).encode(
x=alt.X('iconic_taxon:N', sort='-y', title='Iconic taxon'),
y=alt.Y('count:Q', title='Number of species/observations'), url='img')
bar1 + bar2 + imgiNaturalist group species into iconic taxa. Since we don’t need to get into the details of taxonomic classifications for this project, this will do for this excercise.
For the spatial visualisation of the data, this code brings together the information from the NSW base map, the Terrestrial Vegetation map, the Gayini wetlands boundary polygon and the iNaturalist observations.
m = folium.Map(location=[-34.65, 143.583333],tiles = None, zoom_start=9)
folium.TileLayer(tiles=NSW_basemap,
attr=nsw_base_attrib,
name='NSW base map').add_to(m)
folium.WmsTileLayer(
url=vegmap_wms.provider.url,
name='Vegetation Map',
styles='default',
fmt="image/png",
transparent=True,
layers=vegmap_layer,
attr=vegmap_attrib,
overlay=True,
show=False,
).add_to(m)
geo_j = folium.GeoJson(data=gayini_geojson, style_function=lambda x: {"fillColor": "orange"})
folium.Popup('Gayini wetlands').add_to(geo_j)
pol = folium.FeatureGroup(name="Boundaries", control=True).add_to(m)
geo_j.add_to(pol)
fg = folium.FeatureGroup(name="iNaturalist observations", control=True, attribution="observers @ iNaturalist").add_to(m)
popup_text = """<img src='{url}'>
<caption><i>{species}</i> observed on {observed_on} / {attribution}</caption> {desc}
"""
for obs in observations['results']:
if obs['quality_grade'] == 'research':
if obs['description'] is None:
desc = ""
else:
desc = obs['description']
pincolor = 'green'
else:
desc = "Observation is not research quality grade."
pincolor = 'gray'
fg.add_child(
folium.Marker(
location=obs['location'],
popup=popup_text.format(
species=obs['species_guess'],
observed_on=obs['observed_on'],
desc=desc,
url = obs['observation_photos'][0]['photo']['url'],
attribution = obs['observation_photos'][0]['photo']['attribution']),
icon=folium.Icon(color=pincolor),
)
)
folium.LayerControl().add_to(m)
m.fit_bounds(m.get_bounds())
mLegend for the vegetation map:
display(vegmap_legend)
As the iNaturalist project has started recently, there is still a small number of observations that meet the research quality grade. Gayini is a great place to observe wildlife, but is also a remote place with few visitor uploading data to iNaturalist.
As the NNTC rangers are trained in the use of the iNat app, we expect to see an increase in the number of records and species detected.
The aim of this code is to be re-used and adapted to track the progress of the project.
I acknowledge the Bedegal and Gadigal peoples (Canberra) who are the Traditional Owners of the lands where UNSW Sydney is located, and the Nari-Nari who are the Traditional Custodians of the Gayini Wetlands.
This work was supported by the Ian Potter Foundation and ARDC Gayini TEDISC project.
This document was created with quarto, Jupyter, Python, and good quality coffee.
See the code tools in the top right corner of this document for all the source code, and the citation information at the bottom of this document.
import session_info
session_info.show()----- PIL 11.1.0 altair 5.5.0 folium 0.19.4 geopandas 1.0.1 owslib 0.32.0 pandas 2.1.1 pyinaturalist 0.20.1 session_info 1.0.0 -----
anyio NA appnope 0.1.4 arrow 1.3.0 asttokens NA attr 23.1.0 attrs 23.1.0 babel 2.16.0 backports NA branca 0.6.0 cattr NA cattrs NA certifi 2023.07.22 charset_normalizer 3.2.0 comm 0.2.2 cycler 0.10.0 cython_runtime NA dateutil 2.8.2 debugpy 1.8.12 decorator 5.1.1 defusedxml 0.7.1 executing 2.1.0 fastjsonschema NA fqdn NA google NA idna 3.4 importlib_metadata NA ipykernel 6.29.5 isoduration NA itsdangerous 2.2.0 jaraco NA jedi 0.19.2 jinja2 3.1.2 json5 0.10.0 jsonpointer 3.0.0 jsonschema 4.23.0 jsonschema_specifications NA jupyter_events 0.11.0 jupyter_server 2.15.0 jupyterlab_server 2.27.3 keyring NA kiwisolver 1.4.5 lxml 5.3.0 markupsafe 2.1.3 matplotlib 3.8.0 matplotlib_inline 0.1.7 more_itertools 10.6.0 mpl_toolkits NA narwhals 1.22.0 nbformat 5.10.4 numpy 1.26.0 overrides NA packaging 23.1 parso 0.8.4 platformdirs 4.3.6 plotly 5.24.1 prometheus_client NA prompt_toolkit 3.0.48 psutil 6.1.1 pure_eval 0.2.3 pydev_ipython NA pydevconsole NA pydevd 3.2.3 pydevd_file_utils NA pydevd_plugins NA pydevd_tracing NA pygments 2.19.1 pyparsing 3.1.1 pyproj 3.6.1 pyrate_limiter NA pythonjsonlogger NA pytz 2023.3.post1 referencing NA requests 2.31.0 requests_cache 1.2.1 requests_ratelimiter NA rfc3339_validator 0.1.4 rfc3986_validator 0.1.1 rich NA rpds NA send2trash NA shapely 2.0.2 simplejson 3.19.3 sitecustomize NA six 1.16.0 sniffio 1.3.0 socks 1.7.1 stack_data 0.6.3 tenacity NA tornado 6.4.2 traitlets 5.14.3 typing_extensions NA uri_template NA url_normalize 1.4.3 urllib3 1.26.16 wcwidth 0.2.13 webcolors NA websocket 1.8.0 xyzservices 2025.1.0 yaml 6.0.2 zipp NA zmq 26.2.0
----- IPython 8.31.0 jupyter_client 8.6.3 jupyter_core 5.7.2 jupyterlab 4.3.4 notebook 7.3.2 ----- Python 3.11.9 (main, Apr 2 2024, 08:25:04) [Clang 15.0.0 (clang-1500.3.9.4)] macOS-15.3.1-arm64-arm-64bit ----- Session information updated at 2025-02-19 16:26
@online{ferrer-paris2025,
author = {Ferrer-Paris, José R.},
title = {Reproducible Workflow for Visualisation of {iNaturalist}
Observations in the {Gayini} Wetlands},
date = {2025-02-19},
url = {https://jrfep.quarto.pub/gayini-inat/},
langid = {en}
}