Differences

This shows you the differences between two versions of the page.

--- processing:workflow_training [2019/02/19 11:58] – smerkel
+++ processing:workflow_training [2019/09/04 18:47] (current) – ↷ Links adapted because of a move operation smerkel
@@ Line 11: / Line 11: @@
   - have a look at the generated diffraction images with [[software:fabian|Fabian]]
   - [[processing:search_for_peaks|search for peaks]] using [[software:peaksearch|PeakSearch]]
-  - adjust your experimental parameters and evaluate g-vectors in [[software:imaged11|ImageD11]]
+  - adjust your experimental parameters and [[processing:compute_gvectors|evaluate g-vectors]] in [[software:imaged11|ImageD11]]
   - index your list of extracted g-vectors, using [[software:grainspotter|GrainSpotter]], for instance.
   - [[processing:compare-grain-orientations|compare the indexed grains]] with those generated at the beginning of the procedure.
@@ Line 17: / Line 17: @@
 ===== Details =====
-<WRAP center round tip 60%>
+<WRAP center round tip 80%>
 After following this section, you should understand the parameters involved with multigrain diffraction data processing. You should train with virtual data, assigning rotation ranges, steps, experimental geometry, and crystal structures similar to those of your experiments. At the end, you should be able to re-index 80 to 90% of the starting grains, with very little erroneous indexings.
 </WRAP>
-== Producing data by simulation ==
+==== Producing data by simulation ====
-The purpose of this step is to simulate the outcome of a DAC experiment (grains with random orientations, random strains etc.). It is a possibility to check outcome of an actual experiment. You can also check if your workflow (e.g. the following steps below) is actually working since in a simulation the input parameters are known. When you have real diffraction data already, you can skip this step and open Fabian instead (see below).
+This step will [[processing:synthetic_dataset|simulate the outcome of a the experiment]], with a given number of grains, distribution of orientations, strains, etc.
-The simulation will not only provide 2D diffraction images but also G-vectors, inverse UB matrices (UBi) and some more files. You will get at least 7 different files from the simulation plus the diffraction images. The amount of diffraction images depends on the ω range and the step size you put into the input file. For example, a ω range from -28° to +28° with a step size of 0.5° produces 112 images (numbers from 0 to 111).
+The simulation will not only provide simulated 2D diffraction images but also a list grains and their G-vectors in [[processing:synthetic_dataset#output_files|various formats]]. The amount of diffraction images depends on the ω range and the step size you put choose. For example, an ω range from -28° to +28° with a step size of 0.5° produces 112 images numbered from 0 to 111.
-Create an input file with the ending //.inp//. For a start, simply modify an existing one like [[fileformat:inp:basic|this]]. Afterwards, you can run the simulation with [[software:polyxsim|PolyXSim]]. Write the following to the Konsole:
+Typically, at the end of the simulation, we work with
-  PolyXSim.py -i 'some_input_file'.inp
+  * the generated diffraction images, which we will try to process,
+  * the generated list of grains (in //.gff// format), which we will compare to our indexing.
-The 7 different files (which were just mentioned above) are usually created quite fast. The time consuming process is the creation of images. This time highly depends on the parameters you put in the input file, e.g. the amount of grains, the peak shape and if you switched on strain tensors or noise. If you just want to test if the software is working it is wise to use an input file with very simple parameters (only 1 grain, no strain tensors, no noise, small ω range etc.).
+==== Evaluating the simulated diffraction images ====
-While the simulation is running you can already look at the images, which are already created. For this, open a new tab in the Konsole and open Fabian:
+You should look at the simulated diffraction images with [[software:fabian|Fabian]]. The goal of this step is to
-  fabian.py
+  * locate diffraction peaks,
+  * evaluate their intensity and that of the surrounding background,
+  * understand the [[dac_experiments:geometry|concept of the O-Matrix]].
-This is convenient because you can already see at this point if your simulation works. And in case it does not, you can stop the simulation process right now and you don't need to wait until all images are created, which can take very long time. While you're at it, check also the O-matrix. You find it in Fabian under //Image// --> //Orientation//. Choose the one which is the same as in your input file.
+[[software:fabian|Fabian]] allows you to zoom-in on a peak as well as browse through diffraction images as you increase or decrease ω.
+You will be able to evaluate potential issues with peak overlap. How much rotation in ω can you do before you find an other peak? What is the η-range in which you can safely assign this peak and not its neighbor?
 ==== Working on background ====
-To get rid of the background we now add up all the diffraction images and calculate an average and a median image. Then, every image is subtracted by this average/median image which should remove the background. Of course, if you switched off the background in the previous simulation this process won't change anything. But in case you have real data, this procedure is essential!
-For calculating the average and median you use the python script //median.py//. An alternative way is [[software:imagemath|Image Math]]. For //median.py// type to the command line:
+To get rid of the background we now add up all the diffraction images and calculate an average and a median image. Then, every image is subtracted by this average/median image which should remove the background. Typically, with simulated data, background will not strongly affect your peak search. This point, however, is a good step to perform [[processing:backgroundgeneral|median and average images]] calculations. But in case you have real data, this procedure is essential!
-  median.py -h
-The help will pop up and tell you how to use it.
-The calculation will create an additional //.edf// file.
+The average image is a representation of the data that includes
+  * the background,
+  * the diffraction from the //surrounding matrix// or the //powder portion of the sample// that gives you continuous diffraction rings in the image,
+  * the diffraction from the //sample grains//, that give rise to well-defined diffraction spots.
-Next, the actual images have to be subtracted by one of these three images. Usually the m2 image (median) is used for this, because it is less affected by outliers. Before you do this, make sure you have a separate folder to avoid mixing up the actual data with the processed data! Raw data should never be modified!
+The median image is a representation of the data that includes
+  * the background,
+  * the diffraction from the //surrounding matrix// or the //powder portion of the sample// that gives you continuous diffraction rings in the image.
+The diffraction from the //sample grains//, that give rise to well-defined diffraction spots are removed and **//do not contribute//** to the median image!
-Look at the images in Fabian, go to //Image// --> //Correction// --> //Subtract background// and choose the m2 image. Now every image which is currently loaded (including those which you can access by clicking on //next// and //previous//) gets subtracted by this m2 image (median). If it is not simulated data without noise etc. you should see a difference. The peaks should appear clearer.
+In [[software:fabian|Fabian]], you can subtract the median image from the original data. You will see the spots from the sample grains only. In theory, all background and contribution from the surrounding sample matrix should be removed.
 ==== Peak extraction ====
-From these processed images you can now extract the peaks. Look at some random peaks from several images by zooming in (in Fabian) and check out their intensity. Try to estimate a threshold value which defines how intense a peak must be to be seen by the algorithm. Try to define a threshold, which separates peaks from background (everything above the threshold value is a peak, everything below is background). If you are not sure you can also define several threshold values.
+At this point, you are ready with [[processing:search_for_peaks|peak extraction]]. You should test the effect of different thresholds.
-When you defined one (or more) threshold(s) you can start the [[software:peaksearch|PeakSearch]] algorithm:
+Typically, at this step, you will provide the name of the median image. The median image will be subtracted from the data and the thresholds you defined are relative intensities of the peaks, relative to that of the background and contribution of the surrounding sample matrix.
-  peaksearch.py -n ../'directory'/'name stem' -f 'first image number' -l 'last image number' -d ../'directory'/'median.edf file' -t 'threshold value 1' -t 'threshold value 2' ...
-To check the outcome of PeakSearch, you can load the peaks, which were found, into Fabian and see if they match the actual peak positions. To do this, you have to go click on //CrystTools// --> //Peaks// --> //Read peaks// and choose the //.flt// file which PeakSearch just created. They should appear as red circles on the diffraction image. You can switch on/off the diffraction spots by clicking on //CrystTools// --> //Peaks// --> //Show//.
+Evaluate the outcome of the peak search by loading the peaks which were found into [[software:fabian|Fabian]] and see if they match the actual peak positions.
-==== Experimental parameters ====
+==== Evaluate g-vectors ====
-From these peaks you can now fit the experimental parameters. To do this, open [[software:imaged11|ImageD11]] by typing the following to the Konsole:
+The next step in the process is the [[processing:compute_gvectors|calculations of g-vectors]]. For a given reflection in the crystal, **G**<sub>hkl</sub> is perpendicular to the diffracting plane (hkl) and its norm is 2π/d<sub>hkl</sub>. The real space coordinate of **G**<sub>hkl</sub> can be calculated from the experimental measurements, namely the x-ray wavelength λ, the diffraction angle 2θ, the azimuth angle η, and the rotation angle ω.
-  ImageD11_gui.py
-To load the PeakSearch file click on //Transformation// --> //Load filtered peaks// and choose the //.flt// file from the separate folder with the processed data. Although the image is loaded, it is not plotted automatically, because there are two different ways of plotting. One plotting option is the 2D diffraction image which is similar to Fabian (y/z plot). The other possibility is a cake plot (tth/eta plot). Both options can be accessed by clicking on //Transformation//. Note that plotting both options at once is not making sense because the software is using the same scale for both images (which makes it look weird). To switch from one plot to the other just click on the //Clear// button (bottom of the window) and then plot the other one. //Clear// does only erase the plot, the data is still there.
-Before you check the plots you should enter the measurement parameters. Go to //Transformation// --> //Edit parameters// and enter all parameters for your sample. Some of them can be found in the calbration files of the beamline (such as the [[fileformat:poni|poni file]]). Remove all check marks from the vary boxes and press //Ok//.
+In order to do so, you need to precisely evaluate your experimental geometry (beam center, detector distance, detector tilt, etc). Once this is done, g-vectors can be calculated directly from the location peaks extracted from the diffraction images.
-Next you can have a look at the //tth/eta plot//. Most of the peaks should appear to be on imaginary vertical lines. Zoom in and check, if these lines are completely vertical. If not, you might have strain in your sample. If the line looks like a sinus curve of exactly one period this is due to a wrong beam center. To fix this, go back to //Edit parameters// and activate the check marks for the //y-position// and //z-position// of the detector. Press //Ok// and click on //Fit// for several times until the spots don't move anymore. The imaginary lines should now be completely straight (if you don't have strain). If they are not, you can try to fit other parameters.
+Follow the procedure described in the [[processing:compute_gvectors|compute G-vectors]] section. At the end of the procedure, you will save the list of experimentally detected g-vectors in a text [[fileformat:gve|gve]] file.
-At some point you can click on //Transformation// --> //Add unit cell peaks//. Red tick marks will appear which indicate the expected positions of the vertical lines. With this you can check whether your input parameters (cell parameters, detector distance, ...) were correct.
 ==== Grain indexing ====
-This step is necessary to get the G-vectors from your grains.
-In ImageD11, click on //Indexing// --> //Assign peaks to powder rings// (nothing will happen), then click on //Transformation// --> //Compute g-vectors// and finally //Transformation// --> //Save g-vectors//. Make sure the file gets the ending //.gve//.
 To index the grains you need [[software:grainspotter|GrainSpotter]] and an //.ini// file. If you previously did a simulation with [[software:polyxsim|PolyXSim]], you already have an //.ini// file which you can modify for your purposes. Make sure to keep the original and do only modify a copy. For details on what this //.ini// file should contain, check out the [[software:grainspotter|GrainSpotter wiki page]], the [[fileformat:ini|.ini wiki page]] or the [[GrainSpotter manual]]. Make sure the //.ini// file contains the right //.gve// file (the one you just created).
-To start GrainSpotter, type the following to the Konsole:
-  GrainSpotter.0.90 'some_file_name'.ini
-  or
-  grainspotter 'some_file_name'.ini
-For more information on which syntax you should use, check the [[software:grainspotter|GrainSpotter wiki page]].
 The outcome of the GrainSpotter algorithm is three files: a //.gff// file, a //.ubi// file and a //.log// file. These files contain information on the amount of grains it found, their UBi matrices and some more info. If you are already working with real data, you can now interpret what you got.
 ==== Check your workflow ====
@@ Line 102: / Line 94: @@
 If all the simulated UBi matrices match the calculated ones, you can be quite sure that your workflow is running properly. In a next step you can work with real data.