To assist countries in the production of the CPI during the COVID-19 crisis, this website includes: 

  • Brief on general challenges and recommendations on how to ensure the production of the CPI 
  • Guidance and papers on continuing compilation of the CPI 
  • Link to online tool for collecting prices on the web
  • Discussion forum for experts to share questions, experiences and good practices in the "Comments" field at the bottom of the website for registered users. Please follow this link to register.

If you or your organisation have material that could be useful to share you may send this to Carsten.Boldsen@un.org to be posted on the website.

Brief on general challenges and recommendations

Data collection 

Price collection may be restricted due to closed outlets or price collectors may not be allowed to work or enter outlets. It may also be that outlets do not provide the usual set of prices through other channels (e.g. on paper or via e-mail) and/or there may be shortage of staff in the main office to receive and process the prices that are received. Alternative modes of price collection include telephone, e-mail, online prices and scanner data. However, it may be difficult to ensure a minimum coverage of all products (goods and services). In particular, this may be the case for products for which price collectors usually collect prices. This could, for example, be the case for clothing and fresh food in many countries. In such cases the statistical office may have to rely on collecting a minimum of prices for the most important or the most representative products.  

Compilation 

For imputation of missing observations the general recommendation is to follow a bottom-up approach. This means that the first choice is to impute missing prices with observed price developments of similar products or products that are expected to have similar price developments. If such product prices are not available, the next choice will be to impute the missing prices with the average price development of the product group or the elementary aggregate to which the product belong. If these are not available, the closest available higher-level price index should be used for the imputation. 

In some instances it may not be possible to collect prices for specific product groups or elementary aggregates or even indices above the elementary aggregate level. In such cases the price development of the product group or the elementary aggregate may be imputed by the price development of similar product groups or elementary aggregates. If this is not possible, the price development may be imputed by the higher-level index in which the product group or the elementary aggregate enters. However, imputation of a missing elementary aggregate by the overall CPI may also be justified. This corresponds to leaving the elementary aggregate out of the calculation of the CPI. This may be the preferred option if households' expenditures on an elementary aggregate is assessed to be zero or close to zero. In some countries this may be the case for e.g. international travels, domestic airline travels, child care and sports and cultural events.

These are general recommendations. National circumstances and knowledge of the developments for particular markets and products must be considered. In all cases, it is important to apply imputation methods that ensure the index reaches the correct level when again it becomes possible to collect prices and include them in the index. Also, methods and procedures should be documented to assist continuing production of the CPI and for information of users. 

Communication 

It is important to be transparent to ensure the public trust in the CPI. To this end, good practices for dissemination and communication of official statistics should be followed. This implies that expected changes should be communicated to users in advance, including also information about possible delays of the publication of the CPI. Important changes should be documented and communicated to users when the CPI is released. For instance, if imputations are made or products are left out of the calculation, this should be documented. Impacts on the quality or reliability of the CPI may also be mentioned. 

Guidance notes and papers

Consumer Price Index. Continuity Guidance. Intersecretariat Working Group on Price Statistics (IWGPS) (English, Russian)

Consumer Price Index. Continuity Guidance by IWGPS. Presentation slides. UNECE

Guide on producing CPI under lockdown. UNECE (2021)

Guidance on the compilation of the HICP in the context of the COVID-19 crisis. Eurostat

Guidance note on HICP issues emerging from the lifting of lockdown measures. Eurostat

Guidance on the compilation of the house price index and the owner-occupied housing price index during the covid-19 crisis. Eurostat

Measuring the Consumer Price Index during a time of COVID-19. Australian Bureau of Statistics (ABS)

Coronavirus and the effects on UK prices. Plans for data collection, compilation and publication of various prices statistics following movement restrictions as a result of the coronavirus (COVID-19) pandemic. Office for National Statistics, United Kingdom

How to Produce a CPI in a Covid-19 context? The French experience. Marie Leclair, INSEE

Impact of imputation methods on the CPI and HICP in view of the COVID-19 crisis. Koen Link and Antonio Chessa, Statistics Netherlands

Statistics South Africa publishes weekly inflation of essential items during COVID-19 lock down

The impact of Covid-19 on the CPI and HICP compilation – Implications and responses to the case of Albania

Tracking the Covid-19 crisis with high-resolution transaction data. Cambridge Working Papers in Economics. Vasco M. Carvalho, Juan R. Garcia, Stephen Hansen, Álvaro Ortiz, Tomasa Rodrigo, José V. R. Mora and José Ruiz.

Producing the CPI and the COVID-19 pandemic in Latin America and the Caribbean. UNECLAC

Measuring Real Consumption and CPI Bias under Lockdown Conditions. W. Erwin Diewert and Kevin J. Fox. NBER Working paper series, May 2020

Measuring Real Consumption and CPI Bias under Lockdown Conditions. W. Erwin Diewert and Kevin J. Fox. Presentation slides

Compilation of expenditure weights

Guidance on the compilation of HICP weights in case of large changes in consumer expenditures. Eurostat

The 2020 annual re-weight of the Australian Consumer Price Index. Australian Bureau of Statistics

Webscraping 

An introduction to webscraping using R. Randi Johannessen, Statistics Norway

RobotTool for collecting prices on webpages. Statistics Netherlands

RobotTool is the publicly available, free of charge version of the software used by price analyst at Statistics Netherlands to detect price changes of selected products on websites. RobotTool is a semi-automated scraping tool that makes it easier and faster to monitor on-line prices for selected products on the web.


10 Comments

  1. Giorgi Tetraul

    Following the recommendations of the IMF, Geostat is preparing for the April CPI fieldwork. The following groups are expected to be affected the most by the current situation:

    • “Clothing and footwear”
    • “Restaurants and hotels”
    • “Recreation and culture"

    Since not all the hotels/restaurants have a well-developed webpage, their FB pages might be used for contact.

    For other outlets, in our view, business statistics database might be useful to obtain contact information.

    The biggest expected problem is the insufficient number of price observations, resulting in the index bias. 

  2. Robert Cage

    March data collection for the U.S. CPI ended yesterday, so we are beginning our production runs now. Not yet clear what our experience with missing, uncollected price data will be. We anticipate missing price data to be higher than normal for most of our elementary goods and service categories. We think our rent data will be in better shape.  Our estimation system has built-in hierarchical rules for imputation, and also to suppress from publication aggregate indexes that fail to meet a sample sufficiency threshold.  We call this our 'index adequacy criterion' and the rule is essentially this:

    • if a majority (i.e., greater than 50%) of the component (elementary) index cells that define the published aggregate have at least one collected price, then the index can be published; or conversely:
    • If 50% or more of the component cells defining the aggregate are 100% imputed, then the index will automatically be suppressed from publication
    • The rule is adjudicated based on component cell aggregation weights or index relative importance (so its a weighted, as opposed to unweighted, test)
    • In our news release and online data query tool, indexes that fail to meet the adequacy rule will be published as 'n/a' and footnoted as inadequate
    • In a typical month, roughly 20 to 25 of our 8,000+ published index series fail the adequacy rule; we do expect it to be higher for March 2020

    With respect to imputation, there is an interesting conceptual dilemma here. If I am interpreting it correctly, the IMF and UNECE guidance allows for imputation source pools to travel up to the top of the aggregation tree. For example, if data are 100% missing for all food categories, it is permissible to impute food by the all-items index. In the U.S., we stop the imputation source pools within the 'major group' branch of the aggregation tree. This is roughly equivalent to the COICOP two-digit level. And in most cases, we stop the imputation source pool at more granular levels within the major groups. Our imputation method of last-resort is carry-forward, rather than moving higher up on the aggregation tree above 'major-group.'  For example, imagine a scenario where only rent data are collected and all other goods and services are missing. In the U.S., we would not impute the other goods and services by the rent price change. It is possible that rent prices could be in a deflationary state, while the prices of staple goods are increasing rapidly due to acute increases in consumer demand. In this scenario, carry-forward impute would not be a downward bias, but rather use of the rent experience as the imputation source would be. Thoughts?


  3. Giorgi Tetraul

    Referring to Mr. Cage's comment, I would like to make a parallel with Georgian PPI. A couple of years ago we modified our imputation strategy for higher aggregation levels. In particular, if no price is recorded within, say, "Food products", we do not use the next level up imputation (which is "Manufactured products") but the carry-forward method. The reason is exactly what is mentioned in the last paragraph by Mr. Cage - not to use the price development of radically different products as a source for imputation. Therefore, I agree with this logic.

    Although, I think that in the CPI it is more likely that at least some of the 12 major groups will have real indices in most of the countries - "Food and non-alcoholic beverage", "Alcoholic beverages and tobacco", "Transport" (partly), "Health", "Communication" and "Housing, water, electricity, gas and other fuels". Thus, imputation based on the upper-level group method seems more preferable than carry-forward. 


  4. Michael Calabro

    For the discussion on imputation, the Australian Bureau of Statistics view is if it's known there has been zero expenditure or close to zero, the solution is to impute off the headline level CPI. This has the effect of not contributing to the CPI movement for that month. The most common example will be for international holiday/travel where travel restrictions/bans are in place. Keen to hear thoughts if other countries are considering this type of imputation if it's known that there is zero expenditure?

  5. Michael Calabro

    In terms of the Eurostat guidance notes. For international travel, is it correct to view there being a trade off between options 4.1 and 5.1? Option 4.1 more accurately captures the fact there was no expenditure in the month, so doesn't contribute to the headline movement. Option 5.2 preserves the time series, which has a strong seasonal pattern.

  6. fuziah md amin

    According to the CPI manual, it is advised that imputation should be made to the missing items.

    However, there are some outlets that were not operating at all directly during this covid 19. In other words, there is no price transaction occurred. For example hotel services that have been ordered to remain closed during covid 19. So, I would like to ask your opinion on what is the best method to use? Can I assume the price for April is zero?

  7. Giorgi Tetraul

    With reference to the comment by Fuziah md Amin: 

    Putting a zero price would automatically mean providing the service free of charge, which does not match the fact that hotels are closed. In my view, this case should be treated as a temporary closure of an outlet, which happens every month in usual times as well. This approach means imputing the price using the upper-level index. 

    Currently, in Georgia, we are processing the data from April fieldwork. Hotels are closed, therefore, no price was recorded. However, restaurants and hotels are providing a takeaway service, which enabled us to have a real index on this subgroup. Thus, the prices for hotels will be imputed using the price index for catering services. 

    I hope this will be helpful.

    P.S.: for catering services, adjustment is needed to eliminate the cost of delivery service and a special package for food (if used).


  8. Giorgi Tetraul

    Dear colleagues,

    I would like to ask for your opinion on one aspect of the imputation. In particular, what would be the best way to impute prices for municipal transport services.

    Currently, Geostat is working on April inflation data, which is published on Monday, May 4. Due to the stop of operation of metro and buses (which are state-owned, and a travel fee is fixed), which was followed later by the total restriction on cars (including taxi), the only prices that were recorded within the 0.7.3. COICOP code were those for the air transport (following the IMF note, we still collected prices for air tickets, because we usually take the cost of the ticket 1 month prior to the flight, and they were available on websites). Due to the decrease in prices for air tickets, the imputation resulted in a substantial decrease in the total index, because "07.3.1 Passenger transport by railway" and "07.3.2 Passenger transport by road" hold for about 3% of basket weights. This downward effect accounts for approximately -0.5 percentage points.

    In our opinion, for such fixed prices it is better to use the carry-forward method, for two reasons:

    • Fewer fluctuations of the index in the current and next months (when the real prices appear);
    • No need to explain to users why now there is a price decrease, followed by the increase next month (or two months later) in the services that are 100% not subject to change without a special announcement from the government.

    On the other hand, this approach will not take into account that no expenditure was incurred on these services and they will still participate in the index calculation. 

    I look forward to hearing from you. Your advice will be highly appreciated. 


    Best regards, 

    Giorgi

  9. fuziah md amin

    In Malaysia, the largest weight in the basket of municipal  transport services  is air transport.  Prices for all flights have dropped.. 

    Meanwhile, road transport such as buses can only operate at certain time.. The fare is the same as the month before COVID 19

    While railway transport is closed during COVID 19. In order to get the price for rail transport, we believe that the carry forward is the best method 

  10. fuziah md amin

    Dear colleagues,

    The household income and expenditure survey (HIES) takes at least a year to be released. For countries that have implemented this survey in 2019, the findings will be released in 2020. In the same time, in 2020, the updating of the CPI basket will be based on the weighted obtained in 2019 this. But the lockdown has occurred throughout 2020 and may last until 2021. Which is as we know the spending patterns have changed during this lockdown. Is it appropriate to continue updating the basket?

    In additional, the HIES survey was conducted once in every two years. The last time this survey was conducted was in 2017, and the latest was 2019. The next survey is planning to be in 2021.