ARM Logo

ACT Basics


Overview

In this tutorial, you will learn some of the basic features of ACT! We we walk through the different displays you can use, and some of the helpful functionality ACT provides.

Installation

If you don’t have ACT already installed, it can be installed using pip or conda using the commands below. Additional information on installation can be found in the ACT User Guide.

pip install act-atmos

conda install -c conda-forge act-atmos

Some features of ACT are only available if you have some optional dependencies installed. For example, Skew-T plots of radiosonde data will require that MetPy is installed. Additional optional dependencies are listed in ACT’s documentation

Imports

First we are going to start by importing all the necessary python libraries that we need which is just act and matplotlib. We also need glob to deal with files.

import act
import matplotlib.pyplot as plt
import glob

Download ARM Data

Next we are going to download the data we are going to use for this session using the ARM Live Data webservice.

Once you have an ARM account, you should be able to utilize this webservice as well. All you need to do is login to get your token. We have setup a sample token for you in this workshop, shown below!

# Set your own username and token if you have it
username = 'armlive_training'
token = '6413343e8c6a6ade'

# ACT module for downloading data from the ARM web service
results = act.discovery.download_data(username, token, 'sgpmfrsr7nchE11.b1', '2021-03-29', '2021-03-29')
[DOWNLOADING] sgpmfrsr7nchE11.b1.20210329.070000.nc

Reading in a NetCDF File

Congratulations, you just downloaded a file from just the command line! Next up is to read the file into an xarray object using the ACT reader. We then can use Jupyter to print out an interactive listing of everything in the object.

ds = act.io.armfiles.read_netcdf(results)
ds
/Users/mgrover/miniforge3/envs/pyart-docs/lib/python3.10/site-packages/tqdm/auto.py:22: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html
  from .autonotebook import tqdm as notebook_tqdm
<xarray.Dataset>
Dimensions:                                  (time: 4320, bench_angle: 181,
                                              wavelength: 750)
Coordinates:
  * time                                     (time) datetime64[ns] 2021-03-29...
  * bench_angle                              (bench_angle) float32 0.0 ... 180.0
Dimensions without coordinates: wavelength
Data variables: (12/158)
    base_time                                datetime64[ns] 2021-03-29
    time_offset                              (time) datetime64[ns] 2021-03-29...
    hemisp_narrowband_filter1                (time) float32 dask.array<chunksize=(4320,), meta=np.ndarray>
    qc_hemisp_narrowband_filter1             (time) int32 dask.array<chunksize=(4320,), meta=np.ndarray>
    hemisp_narrowband_filter2                (time) float32 dask.array<chunksize=(4320,), meta=np.ndarray>
    qc_hemisp_narrowband_filter2             (time) int32 dask.array<chunksize=(4320,), meta=np.ndarray>
    ...                                       ...
    nominal_calibration_factor_filter5       float32 ...
    nominal_calibration_factor_filter6       float32 ...
    nominal_calibration_factor_filter7       float32 ...
    lat                                      float32 ...
    lon                                      float32 ...
    alt                                      float32 ...
Attributes: (12/38)
    command_line:                mfrsr7nch_ingest -s sgp -f E11
    Conventions:                 ARM-1.2
    process_version:             ingest-mfrsr7nch-1.3-0.el7
    dod_version:                 mfrsr7nch-b1-1.1
    input_source:                /data/collection/sgp/sgpmfrsr7nchE11.00/MFRS...
    site_id:                     sgp
    ...                          ...
    doi:                         10.5439/1429369
    history:                     created by user dsmgr on machine flint at 20...
    _file_dates:                 ['20210329']
    _file_times:                 ['070000']
    _datastream:                 sgpmfrsr7nchE11.b1
    _arm_standards_flag:         1

Clean up the object to CF Standards

In order to utilize all the ACT QC modules, we need to clean up the object to follow Climate and Forecast (CF) standards.

ds.clean.cleanup()
ds
<xarray.Dataset>
Dimensions:                                  (time: 4320, bench_angle: 181,
                                              wavelength: 750)
Coordinates:
  * time                                     (time) datetime64[ns] 2021-03-29...
  * bench_angle                              (bench_angle) float32 0.0 ... 180.0
Dimensions without coordinates: wavelength
Data variables: (12/158)
    base_time                                datetime64[ns] 2021-03-29
    time_offset                              (time) datetime64[ns] 2021-03-29...
    hemisp_narrowband_filter1                (time) float32 dask.array<chunksize=(4320,), meta=np.ndarray>
    qc_hemisp_narrowband_filter1             (time) int32 dask.array<chunksize=(4320,), meta=np.ndarray>
    hemisp_narrowband_filter2                (time) float32 dask.array<chunksize=(4320,), meta=np.ndarray>
    qc_hemisp_narrowband_filter2             (time) int32 dask.array<chunksize=(4320,), meta=np.ndarray>
    ...                                       ...
    nominal_calibration_factor_filter5       float32 ...
    nominal_calibration_factor_filter6       float32 ...
    nominal_calibration_factor_filter7       float32 ...
    lat                                      float32 ...
    lon                                      float32 ...
    alt                                      float32 ...
Attributes: (12/31)
    command_line:                mfrsr7nch_ingest -s sgp -f E11
    Conventions:                 ARM-1.2
    process_version:             ingest-mfrsr7nch-1.3-0.el7
    dod_version:                 mfrsr7nch-b1-1.1
    input_source:                /data/collection/sgp/sgpmfrsr7nchE11.00/MFRS...
    site_id:                     sgp
    ...                          ...
    doi:                         10.5439/1429369
    history:                     created by user dsmgr on machine flint at 20...
    _file_dates:                 ['20210329']
    _file_times:                 ['070000']
    _datastream:                 sgpmfrsr7nchE11.b1
    _arm_standards_flag:         1

First Visualization

Let’s plot up some data to see what we’re working with. For this example, we’ll use diffuse_hemisp_narrowband_filter4

variable = 'diffuse_hemisp_narrowband_filter4'

# Create a plotting display object with 2 plots
display = act.plotting.TimeSeriesDisplay(ds, figsize=(15, 10), subplot_shape=(2,))

# Plot up the diffuse variable in the first plot
display.plot(variable, subplot_index=(0,))

# Plot up a day/night background
display.day_night_background(subplot_index=(0,))

# Plot up the QC variable in the second plot
display.qc_flag_block_plot(variable, subplot_index=(1,))

plt.show()
/Users/mgrover/miniforge3/envs/pyart-docs/lib/python3.10/site-packages/act/utils/datetime_utils.py:132: FutureWarning: Unlike other reduction functions (e.g. `skew`, `kurtosis`), the default behavior of `mode` typically preserves the axis it acts along. In SciPy 1.11.0, this behavior will change: the default value of `keepdims` will become False, the `axis` over which the statistic is taken will be eliminated, and the value None will no longer be accepted. Set `keepdims` to True or False to avoid this warning.
  mode = stats.mode(np.diff(time))
../../_images/act-basics_13_1.png

Filter Data

Let’s try and filter some of those outliers out based on the embedded QC in the files.

# Now lets remove some of these outliers
ds.qcfilter.datafilter(variable, rm_tests=[2, 3], del_qc_var=False)

# And plot the data again
# Create a plotting display object with 2 plots
display = act.plotting.TimeSeriesDisplay(ds, figsize=(15, 10), subplot_shape=(2,))

# Plot up the diffuse variable in the first plot
display.plot(variable, subplot_index=(0,))

# Plot up a day/night background
display.day_night_background(subplot_index=(0,))

# Plot up the QC variable in the second plot
display.qc_flag_block_plot(variable, subplot_index=(1,))

plt.show()
/Users/mgrover/miniforge3/envs/pyart-docs/lib/python3.10/site-packages/act/utils/datetime_utils.py:132: FutureWarning: Unlike other reduction functions (e.g. `skew`, `kurtosis`), the default behavior of `mode` typically preserves the axis it acts along. In SciPy 1.11.0, this behavior will change: the default value of `keepdims` will become False, the `axis` over which the statistic is taken will be eliminated, and the value None will no longer be accepted. Set `keepdims` to True or False to avoid this warning.
  mode = stats.mode(np.diff(time))
../../_images/act-basics_15_1.png

Instrument Specific QC Tests

ACT has a growing library of instrument specific tests such as the fast-fourier transform test to detect shading which was adapted from Alexandrov et al 2007. The adaption is that it is applied in a moving window style approach. Note - Check out the webpage as an example of how we are including references to papers behind the codes

Let’s apply it and see how it compares with the DQR!

# Apply test
ds = act.qc.fft_shading_test(ds, variable=variable)

# Create a plotting display object with 2 plots
display = act.plotting.TimeSeriesDisplay(ds, figsize=(15, 10), subplot_shape=(2,))

# Plot up the diffuse variable in the first plot
display.plot(variable, subplot_index=(0,))

# Plot up a day/night background
display.day_night_background(subplot_index=(0,))

# Plot up the QC variable in the second plot
display.qc_flag_block_plot(variable, subplot_index=(1,))

plt.show()
/Users/mgrover/miniforge3/envs/pyart-docs/lib/python3.10/site-packages/act/utils/datetime_utils.py:132: FutureWarning: Unlike other reduction functions (e.g. `skew`, `kurtosis`), the default behavior of `mode` typically preserves the axis it acts along. In SciPy 1.11.0, this behavior will change: the default value of `keepdims` will become False, the `axis` over which the statistic is taken will be eliminated, and the value None will no longer be accepted. Set `keepdims` to True or False to avoid this warning.
  mode = stats.mode(np.diff(time))
/Users/mgrover/miniforge3/envs/pyart-docs/lib/python3.10/site-packages/act/utils/datetime_utils.py:132: FutureWarning: Unlike other reduction functions (e.g. `skew`, `kurtosis`), the default behavior of `mode` typically preserves the axis it acts along. In SciPy 1.11.0, this behavior will change: the default value of `keepdims` will become False, the `axis` over which the statistic is taken will be eliminated, and the value None will no longer be accepted. Set `keepdims` to True or False to avoid this warning.
  mode = stats.mode(np.diff(time))
../../_images/act-basics_17_1.png

Conclusion

In this tutorial, we have shown you how to download data from ARM’s Data Live web service, visualize the data and QC information, filter data based on the QC, and add new QC tests to the dataset. After all this work, you can easily save the xarray object to a NetCDF file using obj.to_netcdf('filename.nc') and all that data will be saved and usable in Python and ACT.

Please checkout the ACT Github repository for the latest and greatest information, including our documentation which has examples that can be downloading in python or Jupyter Notebook formats.

Second ACT!

But wait, there’s more to ACT that we can explore together or that you can do on your own! These examples are going to be more condensed than the above but should still provide you the insight you need to run and do your own things!

We are going to need some additional libraries to help out though!

Imports

import numpy as np

Wind Roses

# Let's download a month of surface meteorological data from the SGP central facility!
results = act.discovery.download_data(username, token, 'sgpmetE13.b1', '2021-05-01', '2021-05-31')

# Read that data into an object (this will concatenate it all for you)
ds = act.io.armfiles.read_netcdf(results)

# Now we can plot up a wind rose of that entire month's worth of data
windrose = act.plotting.WindRoseDisplay(ds, figsize=(10,8))
windrose.plot('wdir_vec_mean', 'wspd_vec_mean', spd_bins=np.linspace(0, 10, 5))
windrose.axes[0].legend()
plt.show()
[DOWNLOADING] sgpmetE13.b1.20210514.000000.cdf
[DOWNLOADING] sgpmetE13.b1.20210516.000000.cdf
[DOWNLOADING] sgpmetE13.b1.20210519.000000.cdf
[DOWNLOADING] sgpmetE13.b1.20210513.000000.cdf
[DOWNLOADING] sgpmetE13.b1.20210512.000000.cdf
[DOWNLOADING] sgpmetE13.b1.20210520.000000.cdf
[DOWNLOADING] sgpmetE13.b1.20210517.000000.cdf
[DOWNLOADING] sgpmetE13.b1.20210518.000000.cdf
[DOWNLOADING] sgpmetE13.b1.20210522.000000.cdf
[DOWNLOADING] sgpmetE13.b1.20210521.000000.cdf
[DOWNLOADING] sgpmetE13.b1.20210523.000000.cdf
[DOWNLOADING] sgpmetE13.b1.20210515.000000.cdf
[DOWNLOADING] sgpmetE13.b1.20210511.000000.cdf
[DOWNLOADING] sgpmetE13.b1.20210502.000000.cdf
[DOWNLOADING] sgpmetE13.b1.20210501.000000.cdf
[DOWNLOADING] sgpmetE13.b1.20210509.000000.cdf
[DOWNLOADING] sgpmetE13.b1.20210505.000000.cdf
[DOWNLOADING] sgpmetE13.b1.20210508.000000.cdf
[DOWNLOADING] sgpmetE13.b1.20210510.000000.cdf
[DOWNLOADING] sgpmetE13.b1.20210503.000000.cdf
[DOWNLOADING] sgpmetE13.b1.20210504.000000.cdf
[DOWNLOADING] sgpmetE13.b1.20210506.000000.cdf
[DOWNLOADING] sgpmetE13.b1.20210507.000000.cdf
[DOWNLOADING] sgpmetE13.b1.20210528.000000.cdf
[DOWNLOADING] sgpmetE13.b1.20210529.000000.cdf
[DOWNLOADING] sgpmetE13.b1.20210530.000000.cdf
[DOWNLOADING] sgpmetE13.b1.20210531.000000.cdf
[DOWNLOADING] sgpmetE13.b1.20210525.000000.cdf
[DOWNLOADING] sgpmetE13.b1.20210524.000000.cdf
[DOWNLOADING] sgpmetE13.b1.20210526.000000.cdf
[DOWNLOADING] sgpmetE13.b1.20210527.000000.cdf
../../_images/act-basics_22_1.png

Present Weather Detector Codes

With the MET system at the main site, there’s also a present weather detector (PWD) deployed. This PWD reports the present weather in WMO codes but can be easily decoded using a utility in ACT. With this information, you can make fancy plots like the DQ Office plots for the PWD.

# Let's just use one of the files from the previous example
ds = act.io.armfiles.read_netcdf(results[21])

# Pass it to the function to decode it along with the variable name
ds = act.utils.inst_utils.decode_present_weather(ds, variable='pwd_pw_code_inst')

# We're going to print out the first 10 decoded values that weren't 0
# This shows the utility of also being able to use the built-in xarray
# features like where!
print(list(ds['pwd_pw_code_inst_decoded'].where(ds.pwd_pw_code_inst > 0, drop=True).values[0:10]))
['Rain, not freezing, slight', 'Rain, not freezing, slight', 'Drizzle, not freezing, slight', 'Drizzle, not freezing, slight', 'Drizzle, not freezing, slight', 'Drizzle, not freezing, slight', 'Drizzle, not freezing, slight', 'Drizzle, not freezing, slight', 'Drizzle, not freezing, slight', 'Drizzle, not freezing, slight']

Doppler Lidar Wind Retrievals

This will show you how you can process the doppler lidar PPI scans to produce wind profiles based on Newsom et al 2016.

# We're going to use some test data that already exists within ACT
ds = act.io.armfiles.read_netcdf(act.tests.sample_files.EXAMPLE_DLPPI_MULTI)

# Returns the wind retrieval information in a new object by default
# Note that the default snr_threshold of 0.008 was too high for the first profile
# Reducing it to 0.002 makes it show up but the quality of the data is likely suspect.
wind_ds = act.retrievals.compute_winds_from_ppi(ds, snr_threshold=0.002)

# Plot it up
display = act.plotting.TimeSeriesDisplay(wind_ds)
display.plot_barbs_from_spd_dir('wind_direction', 'wind_speed', invert_y_axis=False)

#Update the x-limits to make sure both wind profiles are shown
display.axes[0].set_xlim([np.datetime64('2019-10-15T11:45'), np.datetime64('2019-10-15T12:30')])

plt.show()
/Users/mgrover/miniforge3/envs/pyart-docs/lib/python3.10/site-packages/act/plotting/plot.py:80: UserWarning: Could not discern datastreamname and dict or tuple were not provided. Using defaultname of act_datastream!
  warnings.warn(
../../_images/act-basics_26_1.png

Radiosonde Plotting and More!

This will take you through how to plot up a Skew-T plot along with a geographic plot of the radiosonde track on a map. Additionally, will run this through a retrieval to calculate the PBL height using the Liu Liang method.

# Import MetPy if possible
import metpy

# Read in sample radiosonde data and plot up a Skew-T
ds = act.io.armfiles.read_netcdf(act.tests.EXAMPLE_SONDE1)

skewt = act.plotting.SkewTDisplay(ds, figsize=(10, 8))
skewt.plot_from_u_and_v('u_wind', 'v_wind', 'pres', 'tdry', 'dp')

plt.show()
../../_images/act-basics_28_0.png
# Now let's plot up the radiosonde path on a map!
display = act.plotting.GeographicPlotDisplay(ds)
display.geoplot(data_field='pres', title='Radiosonde Path')
plt.show()
../../_images/act-basics_29_0.png
# We need to update the units on temperature before running the retrieval
ds.utils.change_units(variables='tdry', desired_unit='degree_Celsius')
obj = act.retrievals.calculate_pbl_liu_liang(ds)
print('Regime = ', obj['pblht_regime_liu_liang'].values, '\nPBL Height = ', int(obj['pblht_liu_liang'].values))
Regime =  NRL 
PBL Height =  950

Look at Data from SAIL

ACT is not just limited in ARM data… you can also work with data from other organizations, such as NOAA!

During the SAIL and SPLASH field campaigns in Colorado, they deployed a NOAA Snow Level Radar, “FMCW”, around the domain. Let’s download and plot some data! In addition to this radar data, we can also read in some Parsivel data, which is a Laser Disdrometer!

Load in the Data

We use the act.discovery module to load in data for the PSL, and remotely read in the data for the Parsivel data.

# Use the ACT downloader to download a file from the
# Kettle Ponds site on 8/01/2022 between 2200 and 2300 UTC.
data_22_utc = act.discovery.download_noaa_psl_data(
    site='kps', instrument='Radar FMCW Moment', startdate='20220801', hour='22')
data_23_utc = act.discovery.download_noaa_psl_data(
    site='kps', instrument='Radar FMCW Moment', startdate='20220801', hour='23')

# Read in the .raw files from both hours. Spectra data are also downloaded.
radar_ds = act.io.noaapsl.read_psl_radar_fmcw_moment([result_22[-1], result_23[-1]])

# Read in the parsivel text files.
url = ['https://downloads.psl.noaa.gov/psd2/data/realtime/DisdrometerParsivel/Stats/kps/2022/213/kps2221322_stats.txt',
       'https://downloads.psl.noaa.gov/psd2/data/realtime/DisdrometerParsivel/Stats/kps/2022/213/kps2221323_stats.txt']
parsivel_ds = act.io.noaapsl.read_psl_parsivel(url)
Downloading dkps2221322a.mom
Downloading hkps2221322a.mom
Downloading kps2221322.raw
Downloading dkps2221323a.mom
Downloading hkps2221323a.mom
Downloading kps2221323.raw
---------------------------------------------------------------------------
NameError                                 Traceback (most recent call last)
Input In [15], in <cell line: 9>()
      5 data_23_utc = act.discovery.download_noaa_psl_data(
      6     site='kps', instrument='Radar FMCW Moment', startdate='20220801', hour='23')
      8 # Read in the .raw files from both hours. Spectra data are also downloaded.
----> 9 radar_ds = act.io.noaapsl.read_psl_radar_fmcw_moment([result_22[-1], result_23[-1]])
     11 # Read in the parsivel text files.
     12 url = ['https://downloads.psl.noaa.gov/psd2/data/realtime/DisdrometerParsivel/Stats/kps/2022/213/kps2221322_stats.txt',
     13        'https://downloads.psl.noaa.gov/psd2/data/realtime/DisdrometerParsivel/Stats/kps/2022/213/kps2221323_stats.txt']

NameError: name 'result_22' is not defined

Visualize using the TimeSeriesDisplay

Now that we have our data, we can visualize it using a TimeSeriesDisplay!

# Create a TimeSeriesDisplay object using both datasets.
display = act.plotting.TimeSeriesDisplay(
    {"NOAA Site KPS PSL Radar FMCW": radar_ds, "NOAA Site KPS Parsivel": parsivel_ds},
    subplot_shape=(2,), figsize=(10, 10))

# Plot PSL Radar followed by the parsivel data.
display.plot('reflectivity_uncalibrated', dsname='NOAA Site KPS PSL Radar FMCW',
             cmap='act_HomeyerRainbow', subplot_index=(0,))
display.plot('number_density_drops', dsname='NOAA Site KPS Parsivel',
             cmap='act_HomeyerRainbow', subplot_index=(1,))
# Adjust ylims of parsivel plot.
display.axes[1].set_ylim([0, 10])
plt.show()

Conclusions and Additional Resources

Within this notebook, we detailed the key functionality of the Atmospheric data Community Toolkit (ACT), and how to go about analyzing different atmospheric and climate datasets using this package.

If you are interested in additional examples, be sure to check out: