Read and write NetCDF with Python

Available modules to read and write NetCDF files: PyNIO, scipy.io.netcdf, NetCDF4. Each of these modules are already installed on Carter and Conte. Simply type 'module load python' in your terminal to load Python v2.7.2.

Reading and writing netcdf files with each of these packages is very similar (particularly PyNIO and scipy.io.netcdf), differing only slightly in their syntax. NetCDF4, however, does have some different features due to the new features in netcdf version 4 (e.g. groups, unlimited dimension, etc.).
Note: PyNIO supports the reading of other formats besides netcdf (e.g. grib/grib2, hdf5, shapefiles, etc.--see documentation for more details)

Import the Nio module
import Nio

To open a file, we use the open_file constructor:

open_file(filepath, mode, options, history, format)

filepath: the full or relative path to data

mode:
    'r' : read only
    'w', 'r+', 'rw', 'a': (write, read and write, append) ->opens an existing file for writing or creates the file if it doesn't exist 
    'c' : creates a file for writing, but throws an error if the file already exists

    Optional argument that affects how data is read/written. Must be called before open_file constructor.

history:
    Optional argument. It is a user specified string that will get appended to the global "history" attribute of the file that is opened for writing.

format:
    If your file name does not have a conventional file extension at the end (e.g. '.nc', '.netcdf', etc.), we can still open the file with Nio by simply specifying the format. For example, you may want to read in and visualize the raw data output from a WRF run which does not have a netcdf file extension:

file = 'wrfout_d01_2036-06-20_06:00:00'
nc = Nio.open_file(file,'r',format='nc')


Note that we are reading the file, denoted by 'r'.


Print an inventory of variables/dimensions/attributes

print nc

Now that you know the variable names within the file, we can access the content either through index notation [:] or the get_value method for scalar variable (that cannot be indexed)

example:
#dimensions
w_e = nc.dimensions['west_east']
s_e = nc.dimensions['south_north']

#variables
lat = nc.variables['XLAT'][0,:,:]
lon = nc.variables['XLONG'][0,:,:]
pcp = nc.variables['RAINNC'][:]

Accessing variable attributes

p_units = nc.variables['RAINNC'].units
print p_units


  • Write NetCDF

1. Open a file to write to

nc = Nio.open_file('filename_to_write.nc','w')

'w':
'c':


2. Create Dimension(s)


create_variable(name, type, dimensions)


3. Create Variable(s)

create_variable(name, type, dimensions)

Name:  a string specifying what you want your variable name to be
Type:
Dimensions:  specified as a list or tuple of dimension names specified in the order you want them to appear

Special case:

4. Assign values

Special case:
assign_value()

5. Write to file and close



scipy.io.netcdf

import scipy.io.netcdf as netcdf

file = ''

  • Read NetCDF

  • Write NetCDF 


1. Open a file to write to

scipy.io.netcdf.netcdf_file(filename, mode='r', mmap=None, version=1)


ncfile = netcdf.netcdf_file('filename_to_write.nc','w')


2. Create Dimension(s)

netcdf_file.createDimension(name, length)


3. Create Variable(s)

netcdf_file.createVariable(name, type, dimensions)

Name:  a string specifying what you want your variable name to be
Type:
Dimensions:  specified as a list or tuple of dimension names specified in the order you want them to appear

Special case: Assign a scalar value to a variable that cannot be indexed
    • only need to specify name and type in this case
    • netcdf_file.createVariable(name, type)


4. Assign values

Special case: 
netcdf_variable.assignValue(value)

5. Write to file and close

nc.flush() or nc.sync()
nc.close()



NetCDF4

NetCDF4 expands upon the NetCDF3 common data format. Every NetCDF4 file is actually an HDF5 file (but not every HDF5 file is a netcdf file). One of the main advantages of NetCDF4 include its support for larger files, unlimited dimensions (e.g. the time dimension), something called groups, and compression ability using zlib. Using Jeff Whittaker's NetCDF4 Python module is slightly different than the two previous methods discussed, but still easy to pick up. 

  • Read NetCDF



from netCDF4 import Dataset


file = 'example_file.nc'
nc = Dataset(file,'r')


  • Write NetCDF

file = 'filename_to_write.nc'
ncfile = Dataset(file,'w',format='NETCDF4_CLASSIC')

#create dimensions
ncfile.createDimension('lat',numlat)
ncfile.createDimension('lon',numlon)
ncfile.createDimension('level',numnewPlevs)
ncfile.createDimension('time',numtime) 

#define variables
latitude = ncfile.createVariable('lats','d',('lat',))
longitude = ncfile.createVariable('lons','d',('lon',))
tv = ncfile.createVariable('Tv','d',('time','level','lat','lon'))
plevo = ncfile.createVariable('P','d',('level',))
longitude[:] = lon
latitude[:] = lat
tv[:] = Tv
plevo[:] = newPlevs

#close ncfile
ncfile.close()



Comments