Import OpenCSD -- an ARM CoreSight Trace Decode library.vendor/opencsd/a1961c91b02a92f3c6ed8b145c636ac4c5565aca

Git ID a1961c91b02a92f3c6ed8b145c636ac4c5565aca

Sponsored by:	DARPA, AFRL

15 files changed, 1553 insertions, 84 deletions

diff --git a/decoder/tests/auto-fdo/autofdo.md b/decoder/tests/auto-fdo/autofdo.md
new file mode 100644
index 000000000000..b1f22417b50e
--- /dev/null
+++ b/decoder/tests/auto-fdo/autofdo.md
@@ -0,0 +1,523 @@
+AutoFDO and ARM Trace   {#AutoFDO}
+=====================
+
+@brief Using CoreSight trace and perf with OpenCSD for AutoFDO.
+
+## Introduction
+
+Feedback directed optimization (FDO, also know as profile guided
+optimization - PGO) uses a profile of a program's execution to guide the
+optmizations performed by the compiler.  Traditionally, this involves
+building an instrumented version of the program, which records a profile of
+execution as it runs.  The instrumentation adds significant runtime
+overhead, possibly changing the behaviour of the program and it may not be
+possible to run the instrumented program in a production environment
+(e.g. where performance criteria must be met).
+
+AutoFDO uses facilities in the hardware to sample the behaviour of the
+program in the production environment and generate the execution profile.
+An improved profile can be obtained by including the branch history
+(i.e. a record of the last branches taken) when generating an instruction
+samples.  On Arm systems, the ETM can be used to generate such records.
+
+The process can be broken down into the following steps:
+
+* Record execution trace of the program
+* Convert the execution trace to instruction samples with branch histories
+* Convert the instruction samples to source level profiles
+* Use the source level profile with the compiler
+
+This article describes how to enable ETM trace on Arm targets running Linux
+and use the ETM trace to generate AutoFDO profiles and compile an optimized
+program.
+
+
+## Execution trace on Arm targets
+
+Debug and trace of Arm targets is provided by CoreSight.  This consists of
+a set of components that allow access to debug logic, record (trace) the
+execution of a processor and route this data through the system, collecting
+it into a store.
+
+To record the execution of a processor, we require the following
+components:
+
+* A trace source.  The core contains a trace unit, called an ETM that emits
+  data describing the instructions executed by the core.
+* Trace links.  The trace data generated by the ETM must be moved through
+  the system to the component that collects the data (sink).  Links
+  include:
+    * Funnels: merge multiple streams of data
+    * FIFOs: buffer data to smooth out bursts
+    * Replicators: send a stream of data to multiple components
+* Sinks.  These receive the trace data and store it or send it to an
+  external device:
+    * ETB: A small circular buffer (64-128 kilobytes) that stores the most
+      recent data
+    * ETR: A larger (several megabytes) buffer that uses system RAM to
+      store data
+    * TPIU: Sends data to an off-chip capture device (e.g. Arm DSTREAM) 
+
+Each Arm SoC design may have a different layout (topology) of components.
+This topology is described to the OS drivers by the platform's devicetree
+or (in future) ACPI firmware.
+
+For application profiling, we need to store several megabytes of data
+within the system, so will use ETR with the capture tool (perf)
+periodically draining the buffer to a file.
+
+Even though we have a large capture buffer, the ETM can still generate a
+lot of data very quickly - typically an ETM will generate ~1 bit of data
+per instruction (depending on the workload), which results in 256Mbytes per
+second for a core running at 2GHz.  This leads to problems storing and
+decoding such large volumes of data.  AutoFDO uses samples of program
+execution, so we can avoid this problem by using the ETM's features to
+only record small slices of execution - e.g. collect ~5000 cycles of data
+every 50M cycles.  This reduces the data rate to a manageable level - a few
+megabytes per minute.  This technique is known as 'strobing'.
+ 
+
+## Enabling trace
+
+### Driver support
+
+To collect ETM trace, the CoreSight drivers must be included in the
+kernel.  Some of the driver support is not yet included in the mainline
+kernel and many targets are using older kernels.  To enable CoreSight trace
+on these targets, Arm have provided backports of the latest CoreSight
+drivers and ETM strobing patch at:
+
+  <http://linux-arm.org/git?p=linux-coresight-backports.git>
+
+This repository can be cloned with:
+
+```
+git clone git://linux-arm.org/linux-coresight-backports.git
+```
+
+You can include these backports in your kernel by either merging the
+appropriate branch using git or generating patches (using `git
+format-patch`).
+
+For 4.9 based kernels, use the `coresight-4.9-etr-etm_strobe` branch:
+
+```
+git merge coresight-4.9-etr-etm_strobe
+```
+
+or
+
+```
+git format-patch --output-directory /output/dir v4.9..coresight-4.9-etr-etm_strobe
+cd my_kernel
+git am /output/dir/*.patch # or patch -p1 /output/dir/*.patch if not using git
+```
+
+For 4.14 based kernels, use the `coresight-4.14-etm_strobe` branch:
+
+```
+git merge coresight-4.14-etm_strobe
+```
+
+or
+
+```
+git format-patch --output-directory /output/dir v4.14..coresight-4.14-etm_strobe
+cd my_kernel
+git am /output/dir/*.patch # or patch -p1 /output/dir/*.patch if not using git
+```
+
+The CoreSight trace drivers must also be enabled in the kernel
+configuration.  This can be done using the configuration menu (`make
+menuconfig`), selecting `Kernel hacking` / `CoreSight Tracing Support` and
+enabling all options, or by setting the following in the configuration
+file:
+
+```
+CONFIG_CORESIGHT=y
+CONFIG_CORESIGHT_LINK_AND_SINK_TMC=y
+CONFIG_CORESIGHT_SINK_TPIU=y
+CONFIG_CORESIGHT_SOURCE_ETM4X=y
+CONFIG_CORESIGHT_DYNAMIC_REPLICATOR=y
+CONFIG_CORESIGHT_STM=y
+CONFIG_CORESIGHT_CATU=y
+```
+
+Compile the kernel for your target in the usual way, e.g.
+
+```
+make ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- 
+```
+
+Each target may have a different layout of CoreSight components.  To
+collect trace into a sink, the kernel drivers need to know which other
+devices need to be configured to route data from the source to the sink.
+This is described in the devicetree (and in future, the ACPI tables).  The
+device tree will define which CoreSight devices are present in the system,
+where they are located and how they are connected together.  The devicetree
+for some platforms includes a description of the platform's CoreSight
+components, but in other cases you may have to ask the platform/SoC vendor
+to supply it or create it yourself (see Appendix: Describing CoreSight in
+Devicetree).
+  
+Once the target has been booted with the devicetree describing the
+CoreSight devices, you should find the devices in sysfs:
+
+```
+# ls /sys/bus/coresight/devices/
+28440000.etm  28540000.etm  28640000.etm  28740000.etm
+28c03000.funnel  28c04000.etf  28c05000.replicator  28c06000.etr
+28c07000.tpiu
+```
+
+### Perf tools
+
+The perf tool is used to capture execution trace, configuring the trace
+sources to generate trace, routing the data to the sink and collecting the
+data from the sink.
+
+Arm recommends to use the perf version corresponding to the kernel running
+on the target.  This can be built from the same kernel sources with
+
+```
+make -C tools/perf ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- 
+```
+
+If the post-processing (`perf inject`) of the captured data is not being
+done on the target, then the OpenCSD library is not required for this build
+of perf.
+
+Trace is captured by collecting the `cs_etm` event from perf.  The sink
+to collect data into is specified as a parameter of this event.  Trace can
+also be restricted to user space or kernel space with 'u' or 'k'
+parameters.  For example:
+
+```
+perf record -e cs_etm/@28c06000.etr/u --per-thread -- /bin/ls
+```
+    
+Will record the userspace execution of '/bin/ls' into the ETR located at
+0x28c06000.  Note the `--per-thread` option is required - perf currently
+only supports trace of a single thread of execution.  CPU wide trace is a
+work in progresss.
+
+
+## Processing trace and profiles
+
+perf is also used to convert the execution trace an instruction profile.
+This requires a different build of perf, using the version of perf from
+Linux v4.17 or later, as the trace processing code isn't included in the
+driver backports.  Trace decode is provided by the OpenCSD library
+(<https://github.com/Linaro/OpenCSD>), v0.9.1 or later.  This is packaged
+for debian testing (install the libopencsd0, libopencsd-dev packages) or
+can be compiled from source and installed. 
+
+The autoFDO tool <https://github.com/google/autofdo> is used to convert the
+instruction profiles to source profiles for the GCC and clang/llvm
+compilers.
+
+
+## Recording and profiling
+
+Once trace collection using perf is working, we can now use it to profile
+an application.
+
+The application must be compiled to include sufficient debug information to
+map instructions back to source lines.  For GCC, use the `-g1` or `-gmlt`
+options.  For clang/llvm, also add the `-fdebug-info-for-profiling` option.
+
+perf identifies the active program or library using the build identifier
+stored in the elf file.  This should be added at link time with the compiler
+flag `-Wl,--build-id=sha1`.
+
+The next step is to record the execution trace of the application using the
+perf tool.  The ETM strobing should be configured before running the perf
+tool.  There are two parameters:
+
+  * window size: A number of CPU cycles (W)
+  * period: Trace is enabled for W cycle every _period_ * W cycles.
+  
+For example, a typical configuration is to use a window size of 5000 cycles
+and a period of 10000 - this will collect 5000 cycles of trace every 50M
+cycles.  With these proof-of-concept patches, the strobe parameters are
+configured via sysfs - each ETM will have `strobe_window` and
+`strobe_period` parameters in `/sys/bus/coresight/devices/NNNNNNNN.etm` and
+these values will have to be written to each (In a future version, this
+will be integrated into the drivers and perf tool).  The `record.sh`
+script in this directory [`<opencsd>/decoder/tests/auto-fdo`] automates this process.
+
+To collect trace from an application using ETM strobing, run:
+
+```
+taskset -c 0 ./record.sh --strobe 5000 10000 28c06000.etr ./my_application arg1 arg2
+```
+
+The taskset command is used to ensure the process stays on the same CPU
+during execution.
+
+The raw trace can be examined using the `perf report` command:
+
+```
+perf report -D -i perf.data --stdio
+```
+
+For example:
+
+```
+0x1d370 [0x30]: PERF_RECORD_AUXTRACE size: 0x2003c0  offset: 0  ref: 0x39ba881d145f8639  idx: 0  tid: 4551  cpu: -1
+
+. ... CoreSight ETM Trace data: size 2098112 bytes
+        Idx:0; ID:12;   I_ASYNC : Alignment Synchronisation.
+        Idx:12; ID:12;  I_TRACE_INFO : Trace Info.; INFO=0x0
+        Idx:17; ID:12;  I_ADDR_L_64IS0 : Address, Long, 64 bit, IS0.; Addr=0xFFFF000008A4991C; 
+        Idx:48; ID:14;  I_ASYNC : Alignment Synchronisation.
+        Idx:60; ID:14;  I_TRACE_INFO : Trace Info.; INFO=0x0
+        Idx:65; ID:14;  I_ADDR_L_64IS0 : Address, Long, 64 bit, IS0.; Addr=0xFFFF000008A4991C; 
+        Idx:96; ID:14;  I_ASYNC : Alignment Synchronisation.
+        Idx:108; ID:14; I_TRACE_INFO : Trace Info.; INFO=0x0
+        Idx:113; ID:14; I_ADDR_L_64IS0 : Address, Long, 64 bit, IS0.; Addr=0xFFFF000008A4991C; 
+        Idx:122; ID:14; I_TRACE_ON : Trace On.
+        Idx:123; ID:14; I_ADDR_CTXT_L_64IS0 : Address & Context, Long, 64 bit, IS0.; Addr=0x0000000000407B00; Ctxt: AArch64,EL0, NS; 
+        Idx:134; ID:14; I_ATOM_F3 : Atom format 3.; ENN
+        Idx:135; ID:14; I_ATOM_F5 : Atom format 5.; NENEN
+        Idx:136; ID:14; I_ATOM_F5 : Atom format 5.; ENENE
+        Idx:137; ID:14; I_ATOM_F5 : Atom format 5.; NENEN
+        Idx:138; ID:14; I_ATOM_F3 : Atom format 3.; ENN
+        Idx:139; ID:14; I_ATOM_F3 : Atom format 3.; NNE
+        Idx:140; ID:14; I_ATOM_F1 : Atom format 1.; E
+.....
+```
+
+The execution trace is then converted to an instruction profile using
+the perf build with trace decode support.  This may be done on a different
+machine than that which collected the trace (e.g. when cross compiling for
+an embedded target).  The `perf inject` command
+decodes the execution trace and generates periodic instruction samples,
+with branch histories:
+
+```
+perf inject -i perf.data -o inj.data --itrace=i100000il
+```
+
+The `--itrace` option configures the instruction sample behaviour:
+
+* `i100000i` generates an instruction sample every 100000 instructions
+  (only instruction count periods are currently supported, future versions
+  may support time or cycle count periods)
+* `l` includes the branch histories on each sample
+* `b` generates a sample on each branch (not used here)
+
+Perf requires the original program binaries to decode the execution trace.
+If running the `inject` command on a different system than the trace was
+captured on, then the binary and any shared libraries must be added to
+perf's cache with:
+
+```
+perf buildid-cache -a /path/to/binary_or_library
+```
+
+`perf report` can also be used to show the instruction samples:
+
+```
+perf report -D -i inj.data --stdio
+.......
+0x1528 [0x630]: PERF_RECORD_SAMPLE(IP, 0x2): 4551/4551: 0x434b98 period: 3093 addr: 0
+... branch stack: nr:64
+.....  0: 0000000000434b58 -> 0000000000434b68 0 cycles  P   0
+.....  1: 0000000000436a88 -> 0000000000434b4c 0 cycles  P   0
+.....  2: 0000000000436a64 -> 0000000000436a78 0 cycles  P   0
+.....  3: 00000000004369d0 -> 0000000000436a60 0 cycles  P   0
+.....  4: 000000000043693c -> 00000000004369cc 0 cycles  P   0
+.....  5: 00000000004368a8 -> 0000000000436928 0 cycles  P   0
+.....  6: 000000000042d070 -> 00000000004368a8 0 cycles  P   0
+.....  7: 000000000042d108 -> 000000000042d070 0 cycles  P   0
+.......
+..... 57: 0000000000448ee0 -> 0000000000448f24 0 cycles  P   0
+..... 58: 0000000000448ea4 -> 0000000000448ebc 0 cycles  P   0
+..... 59: 0000000000448e20 -> 0000000000448e94 0 cycles  P   0
+..... 60: 0000000000448da8 -> 0000000000448ddc 0 cycles  P   0
+..... 61: 00000000004486f4 -> 0000000000448da8 0 cycles  P   0
+..... 62: 00000000004480fc -> 00000000004486d4 0 cycles  P   0
+..... 63: 0000000000448658 -> 00000000004480ec 0 cycles  P   0
+ ... thread: program1:4551
+ ...... dso: /home/root/program1
+.......
+```
+
+The instruction samples produced by `perf inject` is then passed to the
+autofdo tool to generate source level profiles for the compiler.  For
+clang/LLVM:
+
+```
+create_llvm_prof -binary=/path/to/binary -profile=inj.data -out=program.llvmprof
+```
+
+And for GCC:
+
+```
+create_gcov -binary=/path/to/binary -profile=inj.data -gcov_version=1 -gcov=program.gcov
+```
+
+The profiles can be viewed with:
+
+```
+llvm-profdata show -sample program.llvmprof
+```
+
+Or, for GCC:
+
+```
+dump_gcov -gcov_version=1 program.gcov
+```
+
+## Using profile in the compiler
+
+The profile produced by the above steps can then be passed to the compiler
+to optimize the next build of the program.
+
+For GCC, use the `-fauto-profile` option:
+
+```
+gcc -O2 -fauto-profile=program.gcov -o program program.c
+```
+
+For Clang, use the `-fprofile-sample-use` option:
+
+```
+clang -O2 -fprofile-sample-use=program.llvmprof -o program program.c
+```
+
+
+## Summary
+
+The basic commands to run an application and create a compiler profile are:
+
+```
+taskset -c 0 ./record.sh --strobe 5000 10000 28c06000.etr ./my_application arg1 arg2
+perf inject -i perf.data -o inj.data --itrace=i100000il
+create_llvm_prof -binary=/path/to/binary -profile=inj.data -out=program.llvmprof
+```
+
+Use `create_gcov` for gcc.
+
+
+## References
+
+* AutoFDO tool: <https://github.com/google/autofdo>
+* GCC's wiki on autofdo: <https://gcc.gnu.org/wiki/AutoFDO>, <https://gcc.gnu.org/wiki/AutoFDO/Tutorial>
+* Google paper: <https://ai.google/research/pubs/pub45290>
+* CoreSight kernel docs: Documentation/trace/coresight.txt
+
+
+## Appendix: Describing CoreSight in Devicetree
+
+
+Each component has an entry in the device tree that describes its:
+
+* type: The `compatible` field defines which driver to use
+* location: A `reg` defines the component's address and size on the bus
+* clocks: The `clocks` and `clock-names` fields state which clock provides
+  the `apb_pclk` clock.
+* connections to other components: `port` and `ports` field link the
+  component to ports of other components
+
+To create the device tree, some information about the platform is required:
+
+* The memory address of the CoreSight components.  This is the address in
+  the CPU's address space where the CPU can access each CoreSight
+  component.
+* The connections between the components.
+  
+This information can be found in the SoC's reference manual or you may need
+to ask the platform/SoC vendor to supply it.
+
+An ETMv4 source is declared with a section like this:
+
+```
+	etm0: etm@22040000 {
+		compatible = "arm,coresight-etm4x", "arm,primecell";
+		reg = <0 0x22040000 0 0x1000>;
+
+		cpu = <&A72_0>;
+		clocks = <&soc_smc50mhz>;
+		clock-names = "apb_pclk";
+		port {
+			cluster0_etm0_out_port: endpoint {
+				remote-endpoint = <&cluster0_funnel_in_port0>;
+			};
+		};
+	};
+```
+
+This describes an ETMv4 attached to core A72_0, located at 0x22040000, with
+its output linked to port 0 of a funnel.  The funnel is described with:
+
+```
+	funnel@220c0000 { /* cluster0 funnel */
+		compatible = "arm,coresight-funnel", "arm,primecell";
+		reg = <0 0x220c0000 0 0x1000>;
+
+		clocks = <&soc_smc50mhz>;
+		clock-names = "apb_pclk";
+		power-domains = <&scpi_devpd 0>;
+		ports {
+			#address-cells = <1>;
+			#size-cells = <0>;
+
+			port@0 {
+				reg = <0>;
+				cluster0_funnel_out_port: endpoint {
+					remote-endpoint = <&main_funnel_in_port0>;
+				};
+			};
+
+			port@1 {
+				reg = <0>;
+				cluster0_funnel_in_port0: endpoint {
+					slave-mode;
+					remote-endpoint = <&cluster0_etm0_out_port>;
+				};
+			};
+
+			port@2 {
+				reg = <1>;
+				cluster0_funnel_in_port1: endpoint {
+					slave-mode;
+					remote-endpoint = <&cluster0_etm1_out_port>;
+				};
+			};
+		};
+	};
+```
+
+This describes a funnel located at 0x220c0000, receiving data from 2 ETMs
+and sending the merged data to another funnel.  We continue describing
+components with similar blocks until we reach the sink (an ETR):
+
+```
+	etr@20070000 {
+		compatible = "arm,coresight-tmc", "arm,primecell";
+		reg = <0 0x20070000 0 0x1000>;
+		iommus = <&smmu_etr 0>;
+
+		clocks = <&soc_smc50mhz>;
+		clock-names = "apb_pclk";
+		power-domains = <&scpi_devpd 0>;
+		port {
+			etr_in_port: endpoint {
+				slave-mode;
+				remote-endpoint = <&replicator_out_port1>;
+			};
+		};
+	};
+```
+
+Full descriptions of the properties of each component can be found in the
+Linux source at Documentation/devicetree/bindings/arm/coresight.txt.
+The Arm Juno platform's devicetree (arch/arm64/boot/dts/arm) provides an example
+description of CoreSight description.
+
+Many systems include a TPIU for off-chip trace.  While this isn't required
+for self-hosted trace, it should still be included in the devicetree.  This
+allows the drivers to access it to ensure it is put into a disabled state,
+otherwise it may limit the trace bandwidth causing data loss.
diff --git a/decoder/tests/auto-fdo/record.sh b/decoder/tests/auto-fdo/record.sh
new file mode 100755
index 000000000000..16d4ba22db3c
--- /dev/null
+++ b/decoder/tests/auto-fdo/record.sh
@@ -0,0 +1,68 @@
+#!/bin/sh
+
+BUFFER_ETF_A53=ec802000.etf
+BUFFER_ETF_A73=ed002000.etf
+BUFFER_ETF_SYS=ec036000.etf
+BUFFER_ETR=ec033000.etr
+
+OUT_FILE=perf.data
+
+STROBE=
+
+while :; do
+  case $1 in
+    --strobe)
+      STROBE=y
+      WINDOW=$2
+      PERIOD=$3
+      shift 3
+      ;;
+    
+    *)
+      break ;;
+  esac
+done
+
+case $1 in
+  etr)
+    BUFFER=$BUFFER_ETR
+    ;;
+
+  etf-sys)
+    BUFFER=$BUFFER_ETF_SYS
+    ;;
+
+  "")
+    BUFFER=$BUFFER_ETR
+    ;;
+
+  *)
+    BUFFER=$1
+    ;;
+esac
+
+shift 1
+
+case $0 in 
+ /*) F=$0 ;;
+ *) F=$(pwd)/$0 ;;
+esac
+
+SCRIPT_DIR=$(dirname $F)
+
+if [ "$STROBE" ]; then
+  for e in /sys/bus/coresight/devices/*.etm/; do
+    printf "%x" $WINDOW | sudo tee $e/strobe_window > /dev/null
+    printf "%x" $PERIOD | sudo tee $e/strobe_period > /dev/null
+  done
+fi
+
+PERF=$SCRIPT_DIR/perf
+
+export LD_LIBRARY_PATH=$SCRIPT_DIR:$LD_LIBRARY_PATH
+
+sudo LD_LIBRARY_PATH=$SCRIPT_DIR:$LD_LIBRARY_PATH $PERF record $PERF_ARGS -e cs_etm/@$BUFFER/u --per-thread "$@"
+
+sudo chown $(id -u):$(id -g) $OUT_FILE
+
+
diff --git a/decoder/tests/build/linux/c_api_pkt_print_test/makefile b/decoder/tests/build/linux/c_api_pkt_print_test/makefile
index 9fe72477249c..b0b56044e032 100644
--- a/decoder/tests/build/linux/c_api_pkt_print_test/makefile
+++ b/decoder/tests/build/linux/c_api_pkt_print_test/makefile
@@ -33,9 +33,7 @@
 #
 
 CC := $(MASTER_CC)
-CC_FLAGS := $(MASTER_CC_FLAGS) -Wno-switch
 LINKER := $(MASTER_LINKER)	
-LINKER_FLAGS := $(MASTER_LINKER_FLAGS) 
 
 PROG = c_api_pkt_print_test
 
@@ -53,22 +51,21 @@ OBJECTS		=	$(BUILD_DIR)/c_api_pkt_print_test.o
 LIBS		=	-L$(LIB_TARGET_DIR) -l$(LIB_BASE_NAME) -l$(LIB_CAPI_NAME) \
 				-L$(LIB_TEST_TARGET_DIR) -l_echo_test_dcd
 
-all:  build_dir test_app copy_libs
+all:  build_dir copy_libs
 
-test_app: 	$(OBJECTS) $(BIN_TEST_TARGET_DIR)/$(PROG)
+test_app: 	$(BIN_TEST_TARGET_DIR)/$(PROG)
 
 
- $(BIN_TEST_TARGET_DIR)/$(PROG):
+ $(BIN_TEST_TARGET_DIR)/$(PROG): $(OBJECTS)
 			mkdir -p  $(BIN_TEST_TARGET_DIR)
-			$(LINKER) $(LINKER_FLAGS) $(OBJECTS) -Wl,--start-group $(LIBS) -Wl,--end-group -o $(BIN_TEST_TARGET_DIR)/$(PROG)
+			$(LINKER) $(LDFLAGS) $(OBJECTS) -Wl,--start-group $(LIBS) -Wl,--end-group -o $(BIN_TEST_TARGET_DIR)/$(PROG)
 			cp $(LIB_TARGET_DIR)/*.so .
 
 build_dir:
 	mkdir -p $(BUILD_DIR)
 
 .PHONY: copy_libs
-
-copy_libs:
+copy_libs: $(BIN_TEST_TARGET_DIR)/$(PROG)
 	cp $(LIB_TARGET_DIR)/*.so $(BIN_TEST_TARGET_DIR)/.
 
 
@@ -81,12 +78,14 @@ DEPS := $(OBJECTS:%.o=%.d)
 
 ## object compile
 $(BUILD_DIR)/%.o : %.c
-			$(CC) $(CC_FLAGS) $(CC_INCLUDES) -MMD $< -o $@ 
+			$(CC) $(CFLAGS) $(CC_INCLUDES) -MMD $< -o $@ 
 
 #### clean
 .PHONY: clean
 clean :
-	rm -f  $(BIN_TEST_TARGET_DIR)/$(PROG) $(OBJECTS)
-	rm -f  $(DEPS)
+	-rm $(BIN_TEST_TARGET_DIR)/$(PROG) $(OBJECTS)
+	-rm $(DEPS)
+	-rm ./*.so
+	-rmdir $(BUILD_DIR)
 
 # end of file makefile
diff --git a/decoder/tests/build/linux/echo_test_dcd_lib/makefile b/decoder/tests/build/linux/echo_test_dcd_lib/makefile
index 2483170dc0d6..31ca38fe12ed 100644
--- a/decoder/tests/build/linux/echo_test_dcd_lib/makefile
+++ b/decoder/tests/build/linux/echo_test_dcd_lib/makefile
@@ -31,13 +31,8 @@
 #
 
 CC := $(MASTER_CC)
-LINKER := $(MASTER_LINKER)	
 LIB := $(MASTER_LIB)
 
-CC_FLAGS := $(MASTER_CC_FLAGS) -fpic -Wno-switch
-LIB_FLAGS := $(MASTER_LIB_FLAGS)
-LINKER_FLAGS := $(MASTER_LINKER_FLAGS) -shared
-
 LIB_NAME = lib_echo_test_dcd
 
 BUILD_DIR=./$(PLAT_DIR)
@@ -53,21 +48,15 @@ CC_INCLUDES	=	\
 OBJECTS		=	$(BUILD_DIR)/ext_dcd_echo_test.o \
 				$(BUILD_DIR)/ext_dcd_echo_test_fact.o
 
-all:  build_dir  $(OBJECTS) $(LIB_TEST_TARGET_DIR)/$(LIB_NAME).a
+all:  build_dir $(LIB_TEST_TARGET_DIR)/$(LIB_NAME).a
 
-$(LIB_TEST_TARGET_DIR)/$(LIB_NAME).a:
+$(LIB_TEST_TARGET_DIR)/$(LIB_NAME).a: $(OBJECTS)
 	mkdir -p $(LIB_TEST_TARGET_DIR)
-	$(LIB) $(LIB_FLAGS) $(LIB_TEST_TARGET_DIR)/$(LIB_NAME).a $(OBJECTS)
+	$(LIB) $(ARFLAGS) $(LIB_TEST_TARGET_DIR)/$(LIB_NAME).a $(OBJECTS)
 
 build_dir:
 	mkdir -p $(BUILD_DIR)
 
-.PHONY: copy_libs
-
-copy_libs:
-	cp $(LIB_TARGET_DIR)/*.so $(BIN_TEST_TARGET_DIR)/.
-
-
 #### build rules
 ## object dependencies
 DEPS := $(OBJECTS:%.o=%.d)
@@ -76,13 +65,14 @@ DEPS := $(OBJECTS:%.o=%.d)
 
 ## object compile
 $(BUILD_DIR)/%.o : %.c
-			$(CC) $(CC_FLAGS) $(CC_INCLUDES) -MMD $< -o $@ 
+			$(CC) $(CFLAGS) $(CC_INCLUDES) -MMD $< -o $@ 
 
 #### clean
 .PHONY: clean
 clean:
-	rm -f $(OBJECTS)
-	rm -f $(LIB_TEST_TARGET_DIR)/$(LIB_NAME).a
-	rm -f $(DEPS)
+	-rm $(OBJECTS)
+	-rm $(LIB_TEST_TARGET_DIR)/$(LIB_NAME).a
+	-rm $(DEPS)
+	-rmdir $(BUILD_DIR) $(LIB_TEST_TARGET_DIR)
 
 # end of file makefile
diff --git a/decoder/tests/build/linux/mem_buffer_eg/makefile b/decoder/tests/build/linux/mem_buffer_eg/makefile
new file mode 100644
index 000000000000..850ed497dafa
--- /dev/null
+++ b/decoder/tests/build/linux/mem_buffer_eg/makefile
@@ -0,0 +1,90 @@
+########################################################
+# Copyright 2019 ARM Limited. All rights reserved.
+# 
+# Redistribution and use in source and binary forms, with or without modification, 
+# are permitted provided that the following conditions are met:
+# 
+# 1. Redistributions of source code must retain the above copyright notice, 
+# this list of conditions and the following disclaimer.
+# 
+# 2. Redistributions in binary form must reproduce the above copyright notice, 
+# this list of conditions and the following disclaimer in the documentation 
+# and/or other materials provided with the distribution. 
+# 
+# 3. Neither the name of the copyright holder nor the names of its contributors 
+# may be used to endorse or promote products derived from this software without 
+# specific prior written permission. 
+# 
+# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS 'AS IS' AND 
+# ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED 
+# WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. 
+# IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, 
+# INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES 
+# (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; 
+# LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND 
+# ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT 
+# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS 
+# SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. 
+# 
+#################################################################################
+
+########
+# RCTDL - test makefile for snapshot lister test.
+#
+
+CXX := $(MASTER_CXX)
+LINKER := $(MASTER_LINKER)	
+
+PROG = mem-buffer-eg
+
+BUILD_DIR=./$(PLAT_DIR)
+
+VPATH	=	 $(OCSD_TESTS)/source 
+
+CXX_INCLUDES	=	\
+			-I$(OCSD_TESTS)/source \
+			-I$(OCSD_INCLUDE) \
+			-I$(OCSD_TESTS)/snapshot_parser_lib/include
+
+OBJECTS		=	$(BUILD_DIR)/mem_buff_demo.o
+
+LIBS		=	-L$(LIB_TEST_TARGET_DIR) -lsnapshot_parser \
+				-L$(LIB_TARGET_DIR) -l$(LIB_BASE_NAME)
+
+all:  build_dir copy_libs
+
+test_app: $(BIN_TEST_TARGET_DIR)/$(PROG)
+
+
+ $(BIN_TEST_TARGET_DIR)/$(PROG): $(OBJECTS)
+			mkdir -p  $(BIN_TEST_TARGET_DIR)
+			$(LINKER) $(LDFLAGS) $(OBJECTS) -Wl,--start-group $(LIBS) -Wl,--end-group -o $(BIN_TEST_TARGET_DIR)/$(PROG)
+
+build_dir:
+	mkdir -p $(BUILD_DIR)
+
+.PHONY: copy_libs
+copy_libs: $(BIN_TEST_TARGET_DIR)/$(PROG)
+	cp $(LIB_TARGET_DIR)/*.so* $(BIN_TEST_TARGET_DIR)/.
+
+
+
+#### build rules
+## object dependencies
+DEPS := $(OBJECTS:%.o=%.d)
+
+-include $(DEPS)
+
+## object compile
+$(BUILD_DIR)/%.o : %.cpp
+			$(CXX) $(CXXFLAGS) $(CXX_INCLUDES) -MMD $< -o $@
+
+#### clean
+.PHONY: clean
+clean :
+	-rm $(BIN_TEST_TARGET_DIR)/$(PROG) $(OBJECTS)
+	-rm $(DEPS)
+	-rm $(BIN_TEST_TARGET_DIR)/*.so*
+	-rmdir $(BUILD_DIR)
+
+# end of file makefile
diff --git a/decoder/tests/build/linux/snapshot_parser_lib/makefile b/decoder/tests/build/linux/snapshot_parser_lib/makefile
index ae3863f3932b..295bab61780e 100644
--- a/decoder/tests/build/linux/snapshot_parser_lib/makefile
+++ b/decoder/tests/build/linux/snapshot_parser_lib/makefile
@@ -32,15 +32,14 @@
 #
 ########################################################################
 
-CPP := $(MASTER_CPP)
+CXX := $(MASTER_CXX)
 LINKER := $(MASTER_LINKER)
 LIB := $(MASTER_LIB)
 
 # avoid build warnings in donated test code
 WSUPPRESS= -Wno-deprecated-declarations -Wno-unused-variable -Wno-reorder
 
-CPP_FLAGS := $(MASTER_CPP_FLAGS) -fpic -Wno-switch $(WSUPPRESS)
-LIB_FLAGS := $(MASTER_LIB_FLAGS)
+CXXFLAGS += $(WSUPPRESS)
 
 LIB_NAME = libsnapshot_parser
 
@@ -53,7 +52,7 @@ PARSER_INCLUDE=$(PARSER_ROOT)/include
 VPATH=	$(PARSER_SOURCE)
 
 
-CPP_INCLUDES= \
+CXX_INCLUDES= \
 				-I$(PARSER_INCLUDE) \
 				-I$(OCSD_INCLUDE)
 
@@ -64,11 +63,11 @@ OBJECTS=$(BUILD_DIR)/device_info.o \
 		$(BUILD_DIR)/snapshot_reader.o \
 		$(BUILD_DIR)/ss_to_dcdtree.o
 
-all: build_dir $(OBJECTS) $(LIB_TEST_TARGET_DIR)/$(LIB_NAME).a
+all: build_dir $(LIB_TEST_TARGET_DIR)/$(LIB_NAME).a
 
-$(LIB_TEST_TARGET_DIR)/$(LIB_NAME).a:
+$(LIB_TEST_TARGET_DIR)/$(LIB_NAME).a: $(OBJECTS)
 	mkdir -p $(LIB_TEST_TARGET_DIR)
-	$(LIB) $(LIB_FLAGS) $(LIB_TEST_TARGET_DIR)/$(LIB_NAME).a $(OBJECTS)
+	$(LIB) $(ARFLAGS) $(LIB_TEST_TARGET_DIR)/$(LIB_NAME).a $(OBJECTS)
 
 build_dir:
 	mkdir -p $(BUILD_DIR)
@@ -82,11 +81,12 @@ DEPS := $(OBJECTS:%.o=%.d)
 
 ## object compile
 $(BUILD_DIR)/%.o : %.cpp
-	$(CPP) $(CPP_FLAGS) $(CPP_INCLUDES) -MMD $< -o $@
+	$(CXX) $(CXXFLAGS) $(CXX_INCLUDES) -MMD $< -o $@
 
 ### clean
 .PHONY: clean
 clean:
-	rm -f $(OBJECTS)
-	rm -f $(DEPS)
-	rm -f $(LIB_TEST_TARGET_DIR)/$(LIB_NAME).a
+	-rm $(OBJECTS)
+	-rm $(DEPS)
+	-rm $(LIB_TEST_TARGET_DIR)/$(LIB_NAME).a
+	-rmdir $(BUILD_DIR) $(LIB_TEST_TARGET_DIR)
diff --git a/decoder/tests/build/linux/trc_pkt_lister/makefile b/decoder/tests/build/linux/trc_pkt_lister/makefile
index 3447b930c9c9..54ce27d351a6 100644
--- a/decoder/tests/build/linux/trc_pkt_lister/makefile
+++ b/decoder/tests/build/linux/trc_pkt_lister/makefile
@@ -32,10 +32,8 @@
 # RCTDL - test makefile for snapshot lister test.
 #
 
-CPP := $(MASTER_CPP)
-CPP_FLAGS := $(MASTER_CPP_FLAGS) -Wno-switch
+CXX := $(MASTER_CXX)
 LINKER := $(MASTER_LINKER)	
-LINKER_FLAGS := $(MASTER_LINKER_FLAGS) 
 
 PROG = trc_pkt_lister
 
@@ -43,7 +41,7 @@ BUILD_DIR=./$(PLAT_DIR)
 
 VPATH	=	 $(OCSD_TESTS)/source 
 
-CPP_INCLUDES	=	\
+CXX_INCLUDES	=	\
 			-I$(OCSD_TESTS)/source \
 			-I$(OCSD_INCLUDE) \
 			-I$(OCSD_TESTS)/snapshot_parser_lib/include
@@ -53,22 +51,21 @@ OBJECTS		=	$(BUILD_DIR)/trc_pkt_lister.o
 LIBS		=	-L$(LIB_TEST_TARGET_DIR) -lsnapshot_parser \
 				-L$(LIB_TARGET_DIR) -l$(LIB_BASE_NAME)
 
-all:  build_dir test_app copy_libs
+all:  build_dir copy_libs
 
-test_app: 	$(OBJECTS) $(BIN_TEST_TARGET_DIR)/$(PROG)
+test_app: $(BIN_TEST_TARGET_DIR)/$(PROG)
 
 
- $(BIN_TEST_TARGET_DIR)/$(PROG):
+ $(BIN_TEST_TARGET_DIR)/$(PROG): $(OBJECTS)
 			mkdir -p  $(BIN_TEST_TARGET_DIR)
-			$(LINKER) $(LINKER_FLAGS) $(OBJECTS) -Wl,--start-group $(LIBS) -Wl,--end-group -o $(BIN_TEST_TARGET_DIR)/$(PROG)
+			$(LINKER) $(LDFLAGS) $(OBJECTS) -Wl,--start-group $(LIBS) -Wl,--end-group -o $(BIN_TEST_TARGET_DIR)/$(PROG)
 
 build_dir:
 	mkdir -p $(BUILD_DIR)
 
 .PHONY: copy_libs
-
-copy_libs:
-	cp $(LIB_TARGET_DIR)/*.so $(BIN_TEST_TARGET_DIR)/.
+copy_libs: $(BIN_TEST_TARGET_DIR)/$(PROG)
+	cp $(LIB_TARGET_DIR)/*.so* $(BIN_TEST_TARGET_DIR)/.
 
 
 
@@ -80,12 +77,14 @@ DEPS := $(OBJECTS:%.o=%.d)
 
 ## object compile
 $(BUILD_DIR)/%.o : %.cpp
-			$(CPP) $(CPP_FLAGS) $(CPP_INCLUDES) -MMD $< -o $@
+			$(CXX) $(CXXFLAGS) $(CXX_INCLUDES) -MMD $< -o $@
 
 #### clean
 .PHONY: clean
 clean :
-			rm -f  $(BIN_TEST_TARGET_DIR)/$(PROG) $(OBJECTS)
-			rm -f  $(DEPS)
+	-rm $(BIN_TEST_TARGET_DIR)/$(PROG) $(OBJECTS)
+	-rm $(DEPS)
+	-rm $(BIN_TEST_TARGET_DIR)/*.so*
+	-rmdir $(BUILD_DIR)
 
 # end of file makefile
diff --git a/decoder/tests/build/win-vs2015/mem-buffer-eg/mem-buffer-eg.vcxproj b/decoder/tests/build/win-vs2015/mem-buffer-eg/mem-buffer-eg.vcxproj
new file mode 100644
index 000000000000..08f93d47d08c
--- /dev/null
+++ b/decoder/tests/build/win-vs2015/mem-buffer-eg/mem-buffer-eg.vcxproj
@@ -0,0 +1,154 @@
+<?xml version="1.0" encoding="utf-8"?>
+<Project DefaultTargets="Build" ToolsVersion="14.0" xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
+  <ItemGroup Label="ProjectConfigurations">
+    <ProjectConfiguration Include="Debug|Win32">
+      <Configuration>Debug</Configuration>
+      <Platform>Win32</Platform>
+    </ProjectConfiguration>
+    <ProjectConfiguration Include="Release|Win32">
+      <Configuration>Release</Configuration>
+      <Platform>Win32</Platform>
+    </ProjectConfiguration>
+    <ProjectConfiguration Include="Debug|x64">
+      <Configuration>Debug</Configuration>
+      <Platform>x64</Platform>
+    </ProjectConfiguration>
+    <ProjectConfiguration Include="Release|x64">
+      <Configuration>Release</Configuration>
+      <Platform>x64</Platform>
+    </ProjectConfiguration>
+  </ItemGroup>
+  <ItemGroup>
+    <ClCompile Include="..\..\..\source\mem_buff_demo.cpp" />
+  </ItemGroup>
+  <PropertyGroup Label="Globals">
+    <ProjectGuid>{BC090130-2C53-4CF6-8AD4-37BF72B8D01A}</ProjectGuid>
+    <RootNamespace>membuffereg</RootNamespace>
+    <WindowsTargetPlatformVersion>8.1</WindowsTargetPlatformVersion>
+  </PropertyGroup>
+  <Import Project="$(VCTargetsPath)\Microsoft.Cpp.Default.props" />
+  <PropertyGroup Condition="'$(Configuration)|$(Platform)'=='Debug|Win32'" Label="Configuration">
+    <ConfigurationType>Application</ConfigurationType>
+    <UseDebugLibraries>true</UseDebugLibraries>
+    <PlatformToolset>v140</PlatformToolset>
+    <CharacterSet>MultiByte</CharacterSet>
+  </PropertyGroup>
+  <PropertyGroup Condition="'$(Configuration)|$(Platform)'=='Release|Win32'" Label="Configuration">
+    <ConfigurationType>Application</ConfigurationType>
+    <UseDebugLibraries>false</UseDebugLibraries>
+    <PlatformToolset>v140</PlatformToolset>
+    <WholeProgramOptimization>true</WholeProgramOptimization>
+    <CharacterSet>MultiByte</CharacterSet>
+  </PropertyGroup>
+  <PropertyGroup Condition="'$(Configuration)|$(Platform)'=='Debug|x64'" Label="Configuration">
+    <ConfigurationType>Application</ConfigurationType>
+    <UseDebugLibraries>true</UseDebugLibraries>
+    <PlatformToolset>v140</PlatformToolset>
+    <CharacterSet>MultiByte</CharacterSet>
+  </PropertyGroup>
+  <PropertyGroup Condition="'$(Configuration)|$(Platform)'=='Release|x64'" Label="Configuration">
+    <ConfigurationType>Application</ConfigurationType>
+    <UseDebugLibraries>false</UseDebugLibraries>
+    <PlatformToolset>v140</PlatformToolset>
+    <WholeProgramOptimization>true</WholeProgramOptimization>
+    <CharacterSet>MultiByte</CharacterSet>
+  </PropertyGroup>
+  <Import Project="$(VCTargetsPath)\Microsoft.Cpp.props" />
+  <ImportGroup Label="ExtensionSettings">
+  </ImportGroup>
+  <ImportGroup Label="Shared">
+  </ImportGroup>
+  <ImportGroup Label="PropertySheets" Condition="'$(Configuration)|$(Platform)'=='Debug|Win32'">
+    <Import Project="$(UserRootDir)\Microsoft.Cpp.$(Platform).user.props" Condition="exists('$(UserRootDir)\Microsoft.Cpp.$(Platform).user.props')" Label="LocalAppDataPlatform" />
+    <Import Project="..\..\..\..\build\win-vs2015\opencsd.props" />
+  </ImportGroup>
+  <ImportGroup Label="PropertySheets" Condition="'$(Configuration)|$(Platform)'=='Release|Win32'">
+    <Import Project="$(UserRootDir)\Microsoft.Cpp.$(Platform).user.props" Condition="exists('$(UserRootDir)\Microsoft.Cpp.$(Platform).user.props')" Label="LocalAppDataPlatform" />
+    <Import Project="..\..\..\..\build\win-vs2015\opencsd.props" />
+  </ImportGroup>
+  <ImportGroup Label="PropertySheets" Condition="'$(Configuration)|$(Platform)'=='Debug|x64'">
+    <Import Project="$(UserRootDir)\Microsoft.Cpp.$(Platform).user.props" Condition="exists('$(UserRootDir)\Microsoft.Cpp.$(Platform).user.props')" Label="LocalAppDataPlatform" />
+    <Import Project="..\..\..\..\build\win-vs2015\opencsd.props" />
+  </ImportGroup>
+  <ImportGroup Label="PropertySheets" Condition="'$(Configuration)|$(Platform)'=='Release|x64'">
+    <Import Project="$(UserRootDir)\Microsoft.Cpp.$(Platform).user.props" Condition="exists('$(UserRootDir)\Microsoft.Cpp.$(Platform).user.props')" Label="LocalAppDataPlatform" />
+    <Import Project="..\..\..\..\build\win-vs2015\opencsd.props" />
+  </ImportGroup>
+  <PropertyGroup Label="UserMacros" />
+  <PropertyGroup Condition="'$(Configuration)|$(Platform)'=='Debug|Win32'">
+    <OutDir>..\..\..\bin\win$(PlatformArchitecture)\dbg\</OutDir>
+    <IntDir>$(Platform)\$(Configuration)\</IntDir>
+  </PropertyGroup>
+  <PropertyGroup Condition="'$(Configuration)|$(Platform)'=='Release|Win32'">
+    <OutDir>..\..\..\bin\win$(PlatformArchitecture)\dbg\</OutDir>
+    <IntDir>$(Platform)\$(Configuration)\</IntDir>
+  </PropertyGroup>
+  <PropertyGroup Condition="'$(Configuration)|$(Platform)'=='Release|x64'">
+    <OutDir>..\..\..\bin\win$(PlatformArchitecture)\rel\</OutDir>
+  </PropertyGroup>
+  <PropertyGroup Condition="'$(Configuration)|$(Platform)'=='Debug|x64'">
+    <OutDir>..\..\..\bin\win$(PlatformArchitecture)\dbg\</OutDir>
+  </PropertyGroup>
+  <ItemDefinitionGroup Condition="'$(Configuration)|$(Platform)'=='Debug|Win32'">
+    <ClCompile>
+      <WarningLevel>Level3</WarningLevel>
+      <Optimization>Disabled</Optimization>
+      <SDLCheck>
+      </SDLCheck>
+      <AdditionalIncludeDirectories>..\..\..\..\include</AdditionalIncludeDirectories>
+      <PreprocessorDefinitions>WIN32;_DEBUG;_CONSOLE;%(PreprocessorDefinitions)</PreprocessorDefinitions>
+    </ClCompile>
+    <Link>
+      <AdditionalLibraryDirectories>..\..\..\..\lib\win$(PlatformArchitecture)\dbg\;..\..\..\..\tests\lib\win$(PlatformArchitecture)\dbg\</AdditionalLibraryDirectories>
+      <AdditionalDependencies>lib$(LIB_BASE_NAME).lib;kernel32.lib;user32.lib;gdi32.lib;winspool.lib;comdlg32.lib;advapi32.lib;shell32.lib;ole32.lib;oleaut32.lib;uuid.lib;odbc32.lib;odbccp32.lib;%(AdditionalDependencies)</AdditionalDependencies>
+      <GenerateDebugInformation>true</GenerateDebugInformation>
+    </Link>
+  </ItemDefinitionGroup>
+  <ItemDefinitionGroup Condition="'$(Configuration)|$(Platform)'=='Debug|x64'">
+    <ClCompile>
+      <WarningLevel>Level3</WarningLevel>
+      <Optimization>Disabled</Optimization>
+      <SDLCheck>true</SDLCheck>
+      <AdditionalIncludeDirectories>..\..\..\..\include</AdditionalIncludeDirectories>
+      <PreprocessorDefinitions>WIN32;_DEBUG;_CONSOLE;%(PreprocessorDefinitions)</PreprocessorDefinitions>
+    </ClCompile>
+    <Link>
+      <AdditionalLibraryDirectories>..\..\..\..\lib\win$(PlatformArchitecture)\dbg\;..\..\..\..\tests\lib\win$(PlatformArchitecture)\dbg\</AdditionalLibraryDirectories>
+    </Link>
+  </ItemDefinitionGroup>
+  <ItemDefinitionGroup Condition="'$(Configuration)|$(Platform)'=='Release|Win32'">
+    <ClCompile>
+      <WarningLevel>Level3</WarningLevel>
+      <Optimization>MaxSpeed</Optimization>
+      <FunctionLevelLinking>true</FunctionLevelLinking>
+      <IntrinsicFunctions>true</IntrinsicFunctions>
+      <SDLCheck>true</SDLCheck>
+      <AdditionalIncludeDirectories>..\..\..\..\include</AdditionalIncludeDirectories>
+      <PreprocessorDefinitions>WIN32;NDEBUG;_CONSOLE;%(PreprocessorDefinitions)</PreprocessorDefinitions>
+    </ClCompile>
+    <Link>
+      <EnableCOMDATFolding>true</EnableCOMDATFolding>
+      <OptimizeReferences>true</OptimizeReferences>
+      <AdditionalLibraryDirectories>..\..\..\..\lib\win$(PlatformArchitecture)\rel\;..\..\..\..\tests\lib\win$(PlatformArchitecture)\rel\</AdditionalLibraryDirectories>
+    </Link>
+  </ItemDefinitionGroup>
+  <ItemDefinitionGroup Condition="'$(Configuration)|$(Platform)'=='Release|x64'">
+    <ClCompile>
+      <WarningLevel>Level3</WarningLevel>
+      <Optimization>MaxSpeed</Optimization>
+      <FunctionLevelLinking>true</FunctionLevelLinking>
+      <IntrinsicFunctions>true</IntrinsicFunctions>
+      <SDLCheck>true</SDLCheck>
+      <AdditionalIncludeDirectories>..\..\..\..\include</AdditionalIncludeDirectories>
+      <PreprocessorDefinitions>WIN32;NDEBUG;_CONSOLE;%(PreprocessorDefinitions)</PreprocessorDefinitions>
+    </ClCompile>
+    <Link>
+      <EnableCOMDATFolding>true</EnableCOMDATFolding>
+      <OptimizeReferences>true</OptimizeReferences>
+      <AdditionalLibraryDirectories>..\..\..\..\lib\win$(PlatformArchitecture)\rel\;..\..\..\..\tests\lib\win$(PlatformArchitecture)\rel\</AdditionalLibraryDirectories>
+    </Link>
+  </ItemDefinitionGroup>
+  <Import Project="$(VCTargetsPath)\Microsoft.Cpp.targets" />
+  <ImportGroup Label="ExtensionTargets">
+  </ImportGroup>
+</Project>
\ No newline at end of file
diff --git a/decoder/tests/build/win-vs2015/mem-buffer-eg/mem-buffer-eg.vcxproj.filters b/decoder/tests/build/win-vs2015/mem-buffer-eg/mem-buffer-eg.vcxproj.filters
new file mode 100644
index 000000000000..ce99a9eb1d73
--- /dev/null
+++ b/decoder/tests/build/win-vs2015/mem-buffer-eg/mem-buffer-eg.vcxproj.filters
@@ -0,0 +1,22 @@
+<?xml version="1.0" encoding="utf-8"?>
+<Project ToolsVersion="4.0" xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
+  <ItemGroup>
+    <Filter Include="Source Files">
+      <UniqueIdentifier>{4FC737F1-C7A5-4376-A066-2A32D752A2FF}</UniqueIdentifier>
+      <Extensions>cpp;c;cc;cxx;def;odl;idl;hpj;bat;asm;asmx</Extensions>
+    </Filter>
+    <Filter Include="Header Files">
+      <UniqueIdentifier>{93995380-89BD-4b04-88EB-625FBE52EBFB}</UniqueIdentifier>
+      <Extensions>h;hh;hpp;hxx;hm;inl;inc;xsd</Extensions>
+    </Filter>
+    <Filter Include="Resource Files">
+      <UniqueIdentifier>{67DA6AB6-F800-4c08-8B7A-83BB121AAD01}</UniqueIdentifier>
+      <Extensions>rc;ico;cur;bmp;dlg;rc2;rct;bin;rgs;gif;jpg;jpeg;jpe;resx;tiff;tif;png;wav;mfcribbon-ms</Extensions>
+    </Filter>
+  </ItemGroup>
+  <ItemGroup>
+    <ClCompile Include="..\..\..\source\mem_buff_demo.cpp">
+      <Filter>Source Files</Filter>
+    </ClCompile>
+  </ItemGroup>
+</Project>
\ No newline at end of file
diff --git a/decoder/tests/ext_dcd_test_eg/c_api_echo_test/ext_dcd_echo_test.c b/decoder/tests/ext_dcd_test_eg/c_api_echo_test/ext_dcd_echo_test.c
index 32365efb216b..e2ad0454e4e2 100644
--- a/decoder/tests/ext_dcd_test_eg/c_api_echo_test/ext_dcd_echo_test.c
+++ b/decoder/tests/ext_dcd_test_eg/c_api_echo_test/ext_dcd_echo_test.c
@@ -330,7 +330,7 @@ void print_test_cov_results(echo_decoder_t *decoder)
         if (coverage[TEST_COV_MSGLOG_CB] == TEST_RES_OK)    /* check we can use the msg logger for outputting the results */
             lib_cb_LogMsg(p_fns, OCSD_ERR_SEV_ERROR, coverage_message);
         else
-            printf(coverage_message);
+	  printf("%s", coverage_message);
     }
 }
 
diff --git a/decoder/tests/run_pkt_decode_tests.bash b/decoder/tests/run_pkt_decode_tests.bash
new file mode 100755
index 000000000000..56b1cbfd38be
--- /dev/null
+++ b/decoder/tests/run_pkt_decode_tests.bash
@@ -0,0 +1,78 @@
+#!/bin/bash
+#################################################################################
+# Copyright 2018 ARM. All rights reserved.
+#
+# Redistribution and use in source and binary forms, with or without modification, 
+# are permitted provided that the following conditions are met:
+# 
+# 1. Redistributions of source code must retain the above copyright notice, 
+# this list of conditions and the following disclaimer.
+# 
+# 2. Redistributions in binary form must reproduce the above copyright notice, 
+# this list of conditions and the following disclaimer in the documentation 
+# and/or other materials provided with the distribution. 
+# 
+# 3. Neither the name of the copyright holder nor the names of its contributors 
+# may be used to endorse or promote products derived from this software without 
+# specific prior written permission. 
+# 
+# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS 'AS IS' AND 
+# ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED 
+# WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. 
+# IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, 
+# INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES 
+# (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; 
+# LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND 
+# ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT 
+# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS 
+# SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. 
+# 
+#################################################################################
+# OpenCSD library: Test script.
+#
+# Test script to run packet lister on each of the snapshots retained with the repository.
+# No attempt is made to compare output results to previous versions,  (output formatting
+# may change due to bugfix / enhancements) or assess the validity  of the trace output.
+#
+#################################################################################
+
+OUT_DIR=./results
+SNAPSHOT_DIR=./snapshots
+BIN_DIR=./bin/linux64/rel
+
+# directories for tests using full decode
+declare -a test_dirs_decode=( "juno-ret-stck"
+                              "a57_single_step"
+                              "bugfix-exact-match"
+                              "juno-uname-001"
+                              "juno-uname-002"
+                              "juno_r1_1"
+                              "tc2-ptm-rstk-t32"
+                              "trace_cov_a15"
+                              "stm_only"
+                              "stm_only-2"
+                              "stm_only-juno"
+                              "TC2"
+                              "Snowball"
+                              "test-file-mem-offsets"
+                            )
+
+
+echo "Running trc_pkt_lister on snapshot directories."
+
+mkdir -p ${OUT_DIR}
+
+# === test the decode set ===
+export LD_LIBRARY_PATH=${BIN_DIR}/.
+
+for test_dir in "${test_dirs_decode[@]}"
+do
+    echo "Testing $test_dir..."
+    ${BIN_DIR}/trc_pkt_lister -ss_dir "${SNAPSHOT_DIR}/$test_dir" -decode -logfilename "${OUT_DIR}/$test_dir.ppl"
+    echo "Done : Return $?"
+done
+
+# === test the TPIU deformatter ===
+echo "Testing a55-test-tpiu..."
+${BIN_DIR}/trc_pkt_lister -ss_dir "${SNAPSHOT_DIR}/a55-test-tpiu" -dstream_format -o_raw_packed -o_raw_unpacked -logfilename "${OUT_DIR}/a55-test-tpiu.ppl"
+echo "Done : Return $?"
diff --git a/decoder/tests/snapshot_parser_lib/source/ss_to_dcdtree.cpp b/decoder/tests/snapshot_parser_lib/source/ss_to_dcdtree.cpp
index 81ae82ba815c..4eeec732c15b 100644
--- a/decoder/tests/snapshot_parser_lib/source/ss_to_dcdtree.cpp
+++ b/decoder/tests/snapshot_parser_lib/source/ss_to_dcdtree.cpp
@@ -79,15 +79,19 @@ bool CreateDcdTreeFromSnapShot::createDecodeTree(const std::string &SourceName,
         if(m_pReader->getTraceBufferSourceTree(SourceName, tree))
         {
             int numDecodersCreated = 0; // count how many we create - if none then give up.
+            uint32_t formatter_flags = OCSD_DFRMTR_FRAME_MEM_ALIGN;
 
             /* make a note of the trace binary file name + path to ss directory */            
             m_BufferFileName = m_pReader->getSnapShotDir() + tree.buffer_info.dataFileName;
 
             ocsd_dcd_tree_src_t src_format = tree.buffer_info.dataFormat == "source_data" ? OCSD_TRC_SRC_SINGLE : OCSD_TRC_SRC_FRAME_FORMATTED;
 
+            if (tree.buffer_info.dataFormat == "dstream_coresight")
+                formatter_flags = OCSD_DFRMTR_HAS_FSYNCS;
+
             /* create the initial device tree */
             // TBD:     handle syncs / hsyncs data from TPIU
-            m_pDecodeTree = DecodeTree::CreateDecodeTree(src_format,OCSD_DFRMTR_FRAME_MEM_ALIGN); 
+            m_pDecodeTree = DecodeTree::CreateDecodeTree(src_format, formatter_flags);
             if(m_pDecodeTree == 0)
             {
                 LogError("Failed to create decode tree object\n");
@@ -507,15 +511,24 @@ void CreateDcdTreeFromSnapShot::processDumpfiles(std::vector<Parser::DumpDef> &d
     while(it != dumps.end())
     {        
         dumpFilePathName = m_pReader->getSnapShotDir() + it->path;
-        if(!TrcMemAccessorFile::isExistingFileAccessor(dumpFilePathName))
-        {
-            ocsd_err_t err = m_pDecodeTree->addBinFileMemAcc(it->address,OCSD_MEM_SPACE_ANY,dumpFilePathName);
-            if(err != OCSD_OK)
-            {                            
-                std::ostringstream oss;
-                oss << "Failed to create memory accessor for file " << dumpFilePathName << ".";
-                LogError(ocsdError(OCSD_ERR_SEV_ERROR,err,oss.str()));
-            }
+        ocsd_file_mem_region_t region;
+        ocsd_err_t err = OCSD_OK;
+
+        region.start_address = it->address;
+        region.file_offset = it->offset;
+        region.region_size = it->length;
+
+        // ensure we respect optional length and offset parameter and
+        // allow multiple dump entries with same file name to define regions
+        if (!TrcMemAccessorFile::isExistingFileAccessor(dumpFilePathName))
+            err = m_pDecodeTree->addBinFileRegionMemAcc(&region, 1, OCSD_MEM_SPACE_ANY, dumpFilePathName);
+        else
+            err = m_pDecodeTree->updateBinFileRegionMemAcc(&region, 1, OCSD_MEM_SPACE_ANY, dumpFilePathName);
+        if(err != OCSD_OK)
+        {                            
+            std::ostringstream oss;
+            oss << "Failed to create memory accessor for file " << dumpFilePathName << ".";
+            LogError(ocsdError(OCSD_ERR_SEV_ERROR,err,oss.str()));
         }
         it++;
     }
diff --git a/decoder/tests/source/c_api_pkt_print_test.c b/decoder/tests/source/c_api_pkt_print_test.c
index 85bd9bf258b6..02c589e4f275 100644
--- a/decoder/tests/source/c_api_pkt_print_test.c
+++ b/decoder/tests/source/c_api_pkt_print_test.c
@@ -63,12 +63,17 @@
 
 /* path to test snapshots, relative to tests/bin/<plat>/<dbg|rel> build output dir */
 #ifdef _WIN32
-const char *default_path_to_snapshot = "..\\..\\..\\snapshots\\juno_r1_1\\";
-const char *tc2_snapshot = "..\\..\\..\\snapshots\\TC2\\";
+const char *default_base_snapshot_path="..\\..\\..\\snapshots";
+const char *juno_snapshot = "\\juno_r1_1\\";
+const char *tc2_snapshot = "\\TC2\\";
 #else
-const char *default_path_to_snapshot = "../../../snapshots/juno_r1_1/";
-const char *tc2_snapshot = "../../../snapshots/TC2/";
+const char *default_base_snapshot_path = "../../snapshots";
+const char *juno_snapshot = "/juno_r1_1/";
+const char *tc2_snapshot = "/TC2/";
 #endif
+static const char *selected_snapshot;
+static const char *usr_snapshot_path = 0;
+#define MAX_TRACE_FILE_PATH_LEN 512
 
 /* trace data and memory file dump names and values - taken from snapshot metadata */
 const char *trace_data_filename = "cstrace.bin";
@@ -80,6 +85,7 @@ const ocsd_vaddr_t mem_dump_address_tc2=0xC0008000;
 /* test variables - set by command line to feature test API */
 static int using_mem_acc_cb = 0;    /* test the memory access callback function */
 static int use_region_file = 0;     /* test multi region memory files */
+static int using_mem_acc_cb_id = 0; /* test the mem acc callback with trace ID parameter */
 
 /* buffer to handle a packet string */
 #define PACKET_STR_LEN 1024
@@ -114,6 +120,7 @@ static int test_lib_printers = 0;
 static int process_cmd_line(int argc, char *argv[])
 {
     int idx = 1;
+    int len = 0;
 
     while(idx < argc)
     {
@@ -137,13 +144,13 @@ static int process_cmd_line(int argc, char *argv[])
         else if(strcmp(argv[idx],"-etmv3") == 0)
         {
             test_protocol =  OCSD_PROTOCOL_ETMV3;
-            default_path_to_snapshot = tc2_snapshot;
+            selected_snapshot = tc2_snapshot;
             mem_dump_address = mem_dump_address_tc2;
         }
         else if(strcmp(argv[idx],"-ptm") == 0)
         {
             test_protocol =  OCSD_PROTOCOL_PTM;
-            default_path_to_snapshot = tc2_snapshot;
+            selected_snapshot = tc2_snapshot;
             mem_dump_address = mem_dump_address_tc2;
         }
         else if(strcmp(argv[idx],"-stm") == 0)
@@ -156,6 +163,12 @@ static int process_cmd_line(int argc, char *argv[])
             using_mem_acc_cb = 1;
             use_region_file = 0;
         }
+        else if (strcmp(argv[idx], "-test_cb_id") == 0)
+        { 
+            using_mem_acc_cb = 1;
+            use_region_file = 0;
+            using_mem_acc_cb_id = 1;
+        }
         else if(strcmp(argv[idx],"-test_region_file") == 0)
         {
             use_region_file = 1;
@@ -181,6 +194,26 @@ static int process_cmd_line(int argc, char *argv[])
         {
             test_lib_printers = 1;
         }
+        else if(strcmp(argv[idx],"-ss_path") == 0)
+        {
+            idx++;
+            if((idx >= argc) || (strlen(argv[idx]) == 0))
+            {
+                printf("-ss_path: Missing path parameter or zero length\n");
+                return -1;
+            }
+            else
+            {
+                len = strlen(argv[idx]);
+                if(len >  (MAX_TRACE_FILE_PATH_LEN - 32))
+                {
+                    printf("-ss_path: path too long\n");
+                    return -1;
+                }
+                usr_snapshot_path = argv[idx];
+            }
+            
+        }
         else if(strcmp(argv[idx],"-help") == 0)
         {
             return -1;
@@ -199,7 +232,8 @@ static void print_cmd_line_help()
     printf("-decode | -decode_only : full decode + trace packets / full decode packets only (default trace packets only)\n");
     printf("-raw / -raw_packed: print raw unpacked / packed data;\n");
     printf("-test_printstr | -test_libprint : ttest lib printstr callback | test lib based packet printers\n");
-    printf("-test_region_file | -test_cb : mem accessor - test multi region file API | test callback API (default single memory file)\n\n");
+    printf("-test_region_file | -test_cb | -test_cb_id : mem accessor - test multi region file API | test callback API [with trcid] (default single memory file)\n\n");
+    printf("-ss_path <path> : path from cwd to /snapshots/ directory. Test prog will append required test subdir\n");
 }
 
 /************************************************************************/
@@ -211,11 +245,13 @@ static ocsd_mem_space_acc_t dump_file_mem_space = OCSD_MEM_SPACE_ANY;   /* memor
 static long mem_file_size = 0;                /* size of the memory file */
 static ocsd_vaddr_t mem_file_en_address = 0;  /* end address last inclusive address in file. */
 
+/* log the memacc output */
+/* #define LOG_MEMACC_CB */
 
 /* decode memory access using a CallBack function 
 * tests CB API and add / remove mem acc API.
 */
-static uint32_t  mem_acc_cb(const void *p_context, const ocsd_vaddr_t address, const ocsd_mem_space_acc_t mem_space, const uint32_t reqBytes, uint8_t *byteBuffer)
+static uint32_t do_mem_acc_cb(const void *p_context, const ocsd_vaddr_t address, const ocsd_mem_space_acc_t mem_space, const uint8_t trc_id, const uint32_t reqBytes, uint8_t *byteBuffer)
 {
     uint32_t read_bytes = 0;
     size_t file_read_bytes;
@@ -248,9 +284,24 @@ static uint32_t  mem_acc_cb(const void *p_context, const ocsd_vaddr_t address, c
         if(file_read_bytes < read_bytes)
             read_bytes = file_read_bytes;
     }
+#ifdef LOG_MEMACC_CB
+    sprintf(packet_str, "mem_acc_cb(addr 0x%08llX, size %d, trcID 0x%02X)\n", address, reqBytes, trc_id);
+    ocsd_def_errlog_msgout(packet_str);
+#endif
     return read_bytes;
 }
 
+static uint32_t mem_acc_cb(const void *p_context, const ocsd_vaddr_t address, const ocsd_mem_space_acc_t mem_space, const uint32_t reqBytes, uint8_t *byteBuffer)
+{
+    return do_mem_acc_cb(p_context, address, mem_space, 0xff, reqBytes, byteBuffer);
+}
+
+static uint32_t mem_acc_id_cb(const void *p_context, const ocsd_vaddr_t address, const ocsd_mem_space_acc_t mem_space, const uint8_t trc_id, const uint32_t reqBytes, uint8_t *byteBuffer)
+{
+    return do_mem_acc_cb(p_context, address, mem_space, trc_id, reqBytes, byteBuffer);
+}
+
+
 /* Create the memory accessor using the callback function and attach to decode tree */
 static ocsd_err_t create_mem_acc_cb(dcd_tree_handle_t dcd_tree_h, const char *mem_file_path)
 {
@@ -262,8 +313,12 @@ static ocsd_err_t create_mem_acc_cb(dcd_tree_handle_t dcd_tree_h, const char *me
         mem_file_size = ftell(dump_file);
         mem_file_en_address = mem_dump_address + mem_file_size - 1;
 
-        err = ocsd_dt_add_callback_mem_acc(dcd_tree_h,
-            mem_dump_address,mem_file_en_address,dump_file_mem_space,&mem_acc_cb,0);
+        if (using_mem_acc_cb_id)
+            err = ocsd_dt_add_callback_trcid_mem_acc(dcd_tree_h, mem_dump_address, 
+                mem_file_en_address, dump_file_mem_space, &mem_acc_id_cb, 0);
+        else
+            err = ocsd_dt_add_callback_mem_acc(dcd_tree_h, mem_dump_address, 
+                mem_file_en_address, dump_file_mem_space, &mem_acc_cb, 0);
         if(err != OCSD_OK)
         {
             fclose(dump_file);
@@ -290,15 +345,19 @@ static void destroy_mem_acc_cb(dcd_tree_handle_t dcd_tree_h)
 static ocsd_err_t create_test_memory_acc(dcd_tree_handle_t handle)
 {
     ocsd_err_t ret = OCSD_OK;
-    char mem_file_path[512];
+    char mem_file_path[MAX_TRACE_FILE_PATH_LEN];
     uint32_t i0adjust = 0x100;
     int i = 0;
-
+    
     /* region list to test multi region memory file API */
     ocsd_file_mem_region_t region_list[4];
 
     /* path to the file containing the memory image traced - raw binary data in the snapshot  */
-    strcpy(mem_file_path,default_path_to_snapshot);
+    if(usr_snapshot_path != 0)
+        strcpy(mem_file_path,usr_snapshot_path);
+    else
+        strcpy(mem_file_path,default_base_snapshot_path);
+    strcat(mem_file_path,selected_snapshot);
     strcat(mem_file_path,memory_dump_filename);
 
     /* 
@@ -899,10 +958,13 @@ int process_trace_data(FILE *pf)
 int main(int argc, char *argv[])
 {
     FILE *trace_data;
-    char trace_file_path[512];
+    char trace_file_path[MAX_TRACE_FILE_PATH_LEN];
     int ret = 0, i, len;
     char message[512];
 
+    /* default to juno */
+    selected_snapshot = juno_snapshot;
+
     /* command line params */
     if(process_cmd_line(argc,argv) != 0)
     {
@@ -911,8 +973,13 @@ int main(int argc, char *argv[])
     }
     
     /* trace data file path */
-    strcpy(trace_file_path,default_path_to_snapshot);
+    if(usr_snapshot_path != 0)
+        strcpy(trace_file_path,usr_snapshot_path);
+    else
+        strcpy(trace_file_path,default_base_snapshot_path);
+    strcat(trace_file_path,selected_snapshot);
     strcat(trace_file_path,trace_data_filename);
+    printf("opening %s trace data file\n",trace_file_path);
     trace_data = fopen(trace_file_path,"rb");
 
     if(trace_data != NULL)
diff --git a/decoder/tests/source/mem_buff_demo.cpp b/decoder/tests/source/mem_buff_demo.cpp
new file mode 100644
index 000000000000..cacc227e941f
--- /dev/null
+++ b/decoder/tests/source/mem_buff_demo.cpp
@@ -0,0 +1,416 @@
+/*
+* \file     mem_buff_demo.cpp
+* \brief    OpenCSD: using the library with memory buffers for data.
+* 
+* \copyright  Copyright (c) 2018, ARM Limited. All Rights Reserved.
+*/
+
+/*
+* Redistribution and use in source and binary forms, with or without modification,
+* are permitted provided that the following conditions are met:
+*
+* 1. Redistributions of source code must retain the above copyright notice,
+* this list of conditions and the following disclaimer.
+*
+* 2. Redistributions in binary form must reproduce the above copyright notice,
+* this list of conditions and the following disclaimer in the documentation
+* and/or other materials provided with the distribution.
+*
+* 3. Neither the name of the copyright holder nor the names of its contributors
+* may be used to endorse or promote products derived from this software without
+* specific prior written permission.
+*
+* THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS 'AS IS' AND
+* ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
+* WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED.
+* IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT,
+* INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES
+* (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;
+* LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND
+* ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+* (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS
+* SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+*/
+
+/* Example showing techniques to drive library using only memory buffers as input data 
+ * and image data, avoiding file i/o in main processing routines. (File I/O used to 
+ * initially populate buffers but this can be replaced if data is generated by a client 
+ * environment running live.)
+ */
+
+#include <cstdio>
+#include <string>
+#include <iostream>
+#include <sstream>
+#include <cstring>
+
+#include "opencsd.h"              // the library
+
+// uncomment below to use callback function for program memory image
+// #define EXAMPLE_USE_MEM_CALLBACK
+
+/* Input trace buffer */
+static uint8_t *input_trace_data = 0;  
+static uint32_t input_trace_data_size = 0;
+
+/* program memory image for decode */
+static uint8_t *program_image_buffer = 0; // buffer for image data.
+static uint32_t program_image_size = 0;   // size of program image data.
+static ocsd_vaddr_t program_image_address = 0;  // load address on target of program image.
+
+/* a message logger to pass to the error logger / decoder. */
+static ocsdMsgLogger logger;
+
+/* logger callback function - print out error strings */
+class logCallback : public ocsdMsgLogStrOutI
+{
+public:
+    logCallback() {};
+    virtual ~logCallback() {};
+    virtual void printOutStr(const std::string &outStr)
+    {
+        std::cout << outStr.c_str();
+    }
+};
+static logCallback logCB;
+
+/* Decode tree is the main decoder framework - contains the frame unpacker, 
+   packet and trace stream decoders, plus memory image references */
+static DecodeTree *pDecoder = 0; 
+
+/* an error logger - Decode tree registers all components with the error logger
+so that errors can be correctly attributed and printed if required
+*/
+static ocsdDefaultErrorLogger err_log;  
+
+/* callbacks used by the library */
+#ifdef EXAMPLE_USE_MEM_CALLBACK
+// program memory image callback definition
+uint32_t  mem_access_callback_fn(const void *p_context, const ocsd_vaddr_t address, const ocsd_mem_space_acc_t mem_space, const uint32_t reqBytes, uint8_t *byteBuffer);
+#endif
+
+// callback for the decoder output elements
+class DecoderOutputProcessor : public ITrcGenElemIn
+{
+public:
+    DecoderOutputProcessor() {};
+    virtual ~DecoderOutputProcessor() {};
+
+    virtual ocsd_datapath_resp_t TraceElemIn(const ocsd_trc_index_t index_sop,
+        const uint8_t trc_chan_id,
+        const OcsdTraceElement &elem)
+    {
+        // must fully process or make a copy of data in here.
+        // element reference only valid for scope of call. 
+
+        // for the example program we will stringise and print - 
+        // but this is a client program implmentation dependent.
+        std::string elemStr;
+        std::ostringstream oss;
+        oss << "Idx:" << index_sop << "; ID:" << std::hex << (uint32_t)trc_chan_id << "; ";
+        elem.toString(elemStr);
+        oss << elemStr << std::endl;
+        logger.LogMsg(oss.str());
+        return OCSD_RESP_CONT;
+    }
+}; 
+static DecoderOutputProcessor output;
+
+/* for test purposes we are initialising from files, but this could be generated test data as
+ part of a larger program and / or compiled in memory images.
+
+ We have hardcoded in one of the snapshots supplied with the library
+ */
+static int initDataBuffers()
+{
+    FILE *fp;
+    std::string filename;
+    long size;
+
+    /* the file names to create the data buffers */
+#ifdef _WIN32
+    static const char *default_base_snapshot_path = "..\\..\\..\\snapshots";
+    static const char *juno_snapshot = "\\juno_r1_1\\";
+#else
+    static const char *default_base_snapshot_path = "../../../snapshots";
+    static const char *juno_snapshot = "/juno_r1_1/";
+#endif
+
+    /* trace data and memory file dump names and values - taken from snapshot metadata */
+    static const char *trace_data_filename = "cstrace.bin";
+    static const char *memory_dump_filename = "kernel_dump.bin";
+    static ocsd_vaddr_t mem_dump_address = 0xFFFFFFC000081000;
+    
+    /* load up the trace data */
+    filename = default_base_snapshot_path;
+    filename += (std::string)juno_snapshot;
+    filename += (std::string)trace_data_filename;
+
+    fp = fopen(filename.c_str(), "rb");
+    if (!fp)
+        return OCSD_ERR_FILE_ERROR;
+    fseek(fp, 0, SEEK_END);
+    size = ftell(fp);
+    input_trace_data_size = (uint32_t)size;
+    input_trace_data = new (std::nothrow) uint8_t[input_trace_data_size];
+    if (!input_trace_data) {
+        fclose(fp);
+        return OCSD_ERR_MEM;
+    } 
+    rewind(fp);
+    fread(input_trace_data, 1, input_trace_data_size, fp);
+    fclose(fp);
+
+    /* load up a memory image */
+    filename = default_base_snapshot_path;
+    filename += (std::string)juno_snapshot;
+    filename += (std::string)memory_dump_filename;
+
+    fp = fopen(filename.c_str(), "rb");
+    if (!fp)
+        return OCSD_ERR_FILE_ERROR;
+    fseek(fp, 0, SEEK_END);
+    size = ftell(fp);
+    program_image_size = (uint32_t)size;
+    program_image_buffer = new (std::nothrow) uint8_t[program_image_size];
+    if (!program_image_buffer) {
+        fclose(fp);
+        return OCSD_ERR_MEM;
+    }
+    rewind(fp);
+    fread(program_image_buffer, 1, program_image_size, fp);
+    fclose(fp);
+    program_image_address = mem_dump_address;
+    return OCSD_OK;
+}
+
+static ocsd_err_t createETMv4StreamDecoder()
+{
+    ocsd_etmv4_cfg trace_config;
+    ocsd_err_t err = OCSD_OK;
+    EtmV4Config *pCfg = 0;
+    
+    /*
+    * populate the ETMv4 configuration structure with
+    * hard coded values from snapshot .ini files.
+    */
+
+    trace_config.arch_ver = ARCH_V8;
+    trace_config.core_prof = profile_CortexA;
+
+    trace_config.reg_configr = 0x000000C1;
+    trace_config.reg_traceidr = 0x00000010;   /* this is the trace ID -> 0x10, change this to analyse other streams in snapshot.*/
+    trace_config.reg_idr0 = 0x28000EA1;
+    trace_config.reg_idr1 = 0x4100F403;
+    trace_config.reg_idr2 = 0x00000488;
+    trace_config.reg_idr8 = 0x0;
+    trace_config.reg_idr9 = 0x0;
+    trace_config.reg_idr10 = 0x0;
+    trace_config.reg_idr11 = 0x0;
+    trace_config.reg_idr12 = 0x0;
+    trace_config.reg_idr13 = 0x0;
+
+    pCfg = new (std::nothrow) EtmV4Config(&trace_config);
+    if (!pCfg)
+        return OCSD_ERR_MEM;
+        
+    err = pDecoder->createDecoder(OCSD_BUILTIN_DCD_ETMV4I,  /* etm v4 decoder */
+                                   OCSD_CREATE_FLG_FULL_DECODER, /* full trace decode */ 
+                                    pCfg);
+    delete pCfg;
+    return err;
+}
+
+/* Create the decode tree and add the error logger,  stream decoder, memory image data to it.
+   Also register the output callback that processes the decoded trace packets. */
+static ocsd_err_t initialiseDecoder()
+{
+    ocsd_err_t ret = OCSD_OK;
+
+    /* use the creation function to get the type of decoder we want 
+        either OCSD_TRC_SRC_SINGLE : single trace source - not frame formatted
+               OCSD_TRC_SRC_FRAME_FORMATTED :multi source - CoreSight trace frame
+        and set the config flags for operation
+                OCSD_DFRMTR_FRAME_MEM_ALIGN: input data mem aligned -> no syncs
+
+       For this test we create a decode that can unpack frames and is not expecting sync packets.
+     */
+    pDecoder = DecodeTree::CreateDecodeTree(OCSD_TRC_SRC_FRAME_FORMATTED, OCSD_DFRMTR_FRAME_MEM_ALIGN);
+    if (!pDecoder)
+        return OCSD_ERR_MEM;
+
+    /* set up decoder logging - the message logger for output, and the error logger for the library */
+    logger.setLogOpts(ocsdMsgLogger::OUT_STR_CB); /* no IO from the logger, just a string callback. */
+    logger.setStrOutFn(&logCB); /* set the callback - in this example it will go to stdio  but this is up to the implementor. */
+    
+    // for debugging - stdio and file
+//    logger.setLogOpts(ocsdMsgLogger::OUT_FILE | ocsdMsgLogger::OUT_STDOUT);
+    
+    err_log.initErrorLogger(OCSD_ERR_SEV_INFO);
+    err_log.setOutputLogger(&logger); /* pass the output logger to the error logger. */
+
+    pDecoder->setAlternateErrorLogger(&err_log); /* pass the error logger to the decoder, do not use the library version. */
+
+    /* now set up the elements that the decoder needs */
+
+    /* we will decode one of the streams in this example 
+       create a Full decode ETMv4 stream decoder */
+    ret = createETMv4StreamDecoder();
+    if (ret != OCSD_OK)
+        return ret;
+    
+    /* as this has full decode we must supply a memory image. */
+
+    ret = pDecoder->createMemAccMapper();   // the mapper is needed to add code images to.
+    if (ret != OCSD_OK)
+        return ret;
+
+#ifdef EXAMPLE_USE_MEM_CALLBACK
+    // in this example we have a single buffer so we demonstrate how to use a callback.
+    // we are passing the buffer pointer as context as we only have one buffer, but this 
+    // could be a structure that is a list of memory image buffers. Context is entirely
+    // client defined.
+    // Always use OCSD_MEM_SPACE_ANY unless there is a reason to restrict the image to a specific
+    // memory space.
+    pDecoder->addCallbackMemAcc(program_image_address, program_image_address + program_image_size-1,
+        OCSD_MEM_SPACE_ANY,mem_access_callback_fn, program_image_buffer);
+#else
+    // or we can use the built in memory buffer interface - split our one buffer into two to 
+    // demonstrate the addition of multiple regions
+    ocsd_vaddr_t block1_st, block2_st;
+    uint32_t block1_sz, block2_sz;
+    uint8_t *p_block1, *p_block2;
+
+    // break our single buffer into 2 buffers for demo purposes
+    block1_sz = program_image_size / 2;
+    block1_sz &= ~0x3; // align
+    block2_sz = program_image_size - block1_sz;
+    block1_st = program_image_address;  // loaded program memory start address of program
+    block2_st = program_image_address + block1_sz;
+    p_block1 = program_image_buffer;
+    p_block2 = program_image_buffer + block1_sz;
+
+    /* how to add 2 "separate" buffers to the decoder */
+    // Always use OCSD_MEM_SPACE_ANY unless there is a reason to restrict the image to a specific
+    // memory space.
+    ret = pDecoder->addBufferMemAcc(block1_st, OCSD_MEM_SPACE_ANY, p_block1, block1_sz);
+    if (ret != OCSD_OK)
+        return ret;
+
+    ret = pDecoder->addBufferMemAcc(block2_st, OCSD_MEM_SPACE_ANY, p_block2, block2_sz);
+    if (ret != OCSD_OK)
+        return ret;
+#endif
+
+    /* finally we need to provide an output callback to recieve the decoded information */
+    pDecoder->setGenTraceElemOutI(&output);
+    return ret;
+}
+
+/* get rid of the objects we created */
+static void destroyDecoder()
+{
+    delete pDecoder;
+    delete [] input_trace_data;
+    delete [] program_image_buffer;
+}
+
+#ifdef EXAMPLE_USE_MEM_CALLBACK
+/* if we set up to use a callback to access memory image then this is what will be called. */
+/* In this case the client must do all the work in determining if the requested address is in the 
+   memory area. */
+uint32_t  mem_access_callback_fn(const void *p_context, const ocsd_vaddr_t address, 
+    const ocsd_mem_space_acc_t mem_space, const uint32_t reqBytes, uint8_t *byteBuffer)
+{
+    ocsd_vaddr_t buf_end_address = program_image_address + program_image_size - 1;
+    uint32_t read_bytes = reqBytes;
+    
+    /* context should be our memory image buffer - if not return 0 bytes read */
+    if (p_context != program_image_buffer)
+        return 0;
+
+    /* not concerned with memory spaces - assume all global */
+    if ((address < program_image_address) || (address > buf_end_address))
+        return 0;   // requested address not in our buffer.
+
+    // if requested bytes from address more than we have, only read to end of buffer
+    if ((address + reqBytes - 1) > buf_end_address)
+        read_bytes = (uint32_t)(buf_end_address - (address - 1));
+
+    // copy the requested data.
+    memcpy(byteBuffer, program_image_buffer + (address - program_image_address), read_bytes);
+
+    return read_bytes;
+}
+#endif
+
+/* use the decoder to process the global trace data buffer */
+static ocsd_datapath_resp_t processTraceData(uint32_t *bytes_done)
+{
+    /* process in blocks of fixed size. */
+    #define DATA_CHUNK_SIZE 2048
+
+    ocsd_datapath_resp_t resp = OCSD_RESP_CONT;
+    uint32_t block_size, buff_offset, bytes_to_do = input_trace_data_size, bytes_processed;
+    ocsd_trc_index_t index = 0;
+
+    /* process the data in chunks, until either all done or
+    * error occurs.
+    */
+    while ((resp == OCSD_RESP_CONT) && (bytes_to_do))
+    {
+        /* size up a block of input data */
+        block_size = (bytes_to_do >= DATA_CHUNK_SIZE) ? DATA_CHUNK_SIZE : bytes_to_do;
+        buff_offset = input_trace_data_size - bytes_to_do;
+
+        /* push it through the decoder */
+        resp = pDecoder->TraceDataIn(OCSD_OP_DATA, index, block_size, 
+            input_trace_data + buff_offset, &bytes_processed);
+
+        /* adjust counter per bytes processed */
+        bytes_to_do -= bytes_processed;
+        index += bytes_processed;
+    }
+
+    /* if all done then signal end of trace - flushes out any remaining data */
+    if (!bytes_to_do)
+        resp = pDecoder->TraceDataIn(OCSD_OP_EOT, 0, 0, 0, 0);
+
+    /* return amount processed */
+    *bytes_done = input_trace_data_size - bytes_to_do;
+    return resp;
+}
+
+/* main routine - init input data, decode, finish ... */
+int main(int argc, char* argv[])
+{
+    int ret = OCSD_OK;
+    ocsd_datapath_resp_t retd;
+    char msg[256];
+    uint32_t bytes_done;
+
+    /* initialise all the data needed for decode */
+    if ((ret = initDataBuffers()) != OCSD_OK)
+    {
+        logger.LogMsg("Failed to create trace data buffers\n");
+        return ret;
+    }
+    /* initialise a decoder object */
+    if ((ret = initialiseDecoder()) == OCSD_OK)
+    {
+            retd = processTraceData(&bytes_done);
+            if (!OCSD_DATA_RESP_IS_CONT(retd))
+            {
+                ret = OCSD_ERR_DATA_DECODE_FATAL;
+                logger.LogMsg("Processing failed with data error\n");
+            }
+
+        /* get rid of the decoder and print a brief result. */
+        destroyDecoder();
+        sprintf(msg, "Processed %u bytes out of %u\n", bytes_done, input_trace_data_size);
+        logger.LogMsg(msg);
+    }
+    else
+        logger.LogMsg("Failed to create decoder for trace processing\n");
+    return ret;
+}
diff --git a/decoder/tests/source/trc_pkt_lister.cpp b/decoder/tests/source/trc_pkt_lister.cpp
index 854f15817b95..50260a5f8b9b 100644
--- a/decoder/tests/source/trc_pkt_lister.cpp
+++ b/decoder/tests/source/trc_pkt_lister.cpp
@@ -70,6 +70,9 @@ static bool decode = false;
 static bool no_undecoded_packets = false;
 static bool pkt_mon = false;
 static int test_waits = 0;
+static bool dstream_format = false;
+static bool tpiu_format = false;
+static bool has_hsync = false;
 
 int main(int argc, char* argv[])
 {
@@ -180,8 +183,11 @@ void print_help()
     oss << "-ss_dir <dir>       Set the directory path to a trace snapshot\n";
     oss << "-ss_verbose         Verbose output when reading the snapshot\n";
     oss << "\nDecode:\n\n";
-    oss << "-id <n>             Set an ID to list (may be used mutiple times) - default if no id set is for all IDs to be printed\n";
+    oss << "-id <n>             Set an ID to list (may be used multiple times) - default if no id set is for all IDs to be printed\n";
     oss << "-src_name <name>    List packets from a given snapshot source name (defaults to first source found)\n";
+    oss << "-dstream_format     Input is DSTREAM framed.";
+    oss << "-tpiu               Input from TPIU - sync by FSYNC.";
+    oss << "-tpiu_hsync         Input from TPIU - sync by FSYNC and HSYNC.";
     oss << "-decode             Full decode of the packets from the trace snapshot (default is to list undecoded packets only\n";
     oss << "-decode_only        Does not list the undecoded packets, just the trace decode.\n";
     oss << "-o_raw_packed       Output raw packed trace frames\n";
@@ -401,6 +407,19 @@ bool process_cmd_line_opts(int argc, char* argv[])
                     optIdx++;
                 }
             }
+            else if (strcmp(argv[optIdx], "-dstream_format") == 0)
+            {
+                dstream_format = true;
+            }
+            else if (strcmp(argv[optIdx], "-tpiu") == 0)
+            {
+                tpiu_format = true;
+            }
+            else if (strcmp(argv[optIdx], "-tpiu_hsync") == 0)
+            {
+                has_hsync = true;
+                tpiu_format = true;
+            }
             else
             {
                 std::ostringstream errstr;
@@ -488,9 +507,19 @@ void ConfigureFrameDeMux(DecodeTree *dcd_tree, RawFramePrinter **framePrinter)
     if(pDeformatter != 0)
     {
         // configuration - memory alinged buffer
-        uint32_t configFlags = OCSD_DFRMTR_FRAME_MEM_ALIGN;
+        uint32_t configFlags = pDeformatter->getConfigFlags();
+
+        // check for TPIU FSYNC & HSYNC
+        if (tpiu_format) configFlags |= OCSD_DFRMTR_HAS_FSYNCS;
+        if (has_hsync) configFlags |= OCSD_DFRMTR_HAS_HSYNCS;
+        // if FSYNC (& HSYNC) - cannot be mem frame aligned.
+        if (tpiu_format) configFlags &= ~OCSD_DFRMTR_FRAME_MEM_ALIGN;
 
-        pDeformatter->Configure(configFlags);
+        if (!configFlags)
+        {
+            configFlags = OCSD_DFRMTR_FRAME_MEM_ALIGN;
+            pDeformatter->Configure(configFlags);
+        }
         if (outRawPacked || outRawUnpacked)
         {
             if (outRawPacked) configFlags |= OCSD_DFRMTR_PACKED_RAW_OUT;
@@ -554,7 +583,12 @@ void ListTracePackets(ocsdDefaultErrorLogger &err_logger, SnapShotReader &reader
                 // process the file, a buffer load at a time
                 while(!in.eof() && !OCSD_DATA_RESP_IS_FATAL(dataPathResp))
                 {
-                    in.read((char *)&trace_buffer[0],bufferSize);   // load a block of data into the buffer
+                    if (dstream_format)
+                    { 
+                        in.read((char *)&trace_buffer[0], 512 - 8);
+                    }
+                    else
+                        in.read((char *)&trace_buffer[0],bufferSize);   // load a block of data into the buffer
 
                     std::streamsize nBuffRead = in.gcount();    // get count of data loaded.
                     std::streamsize nBuffProcessed = 0;         // amount processed in this buffer.
@@ -597,6 +631,22 @@ void ListTracePackets(ocsdDefaultErrorLogger &err_logger, SnapShotReader &reader
                             dataPathResp = dcd_tree->TraceDataIn(OCSD_OP_FLUSH,0,0,0,0);
                         }
                     }
+
+                    /* dump dstream footers */
+                    if (dstream_format) {
+                        in.read((char *)&trace_buffer[0], 8);
+                        if (outRawPacked)
+                        {
+                            std::ostringstream oss;
+                            oss << "DSTREAM footer [";
+                            for (int i = 0; i < 8; i++)
+                            {
+                                oss << "0x" << std::hex << (int)trace_buffer[i] << " ";
+                            }
+                            oss << "]\n";
+                            logger.LogMsg(oss.str());
+                        }
+                    }
                 }
 
                 // fatal error - no futher processing