[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

*Subject*: RE: [linrad] Linrad*From*: Robert McGwier <rwmcgwier@xxxxxxxxxxxxxxx*Date*: Fri, 27 Jun 2003 14:11:05 +0000

Leif: Thank you very much for your insight and experience. This should be a lot of fun to go through. I was not careful enough when I wrote before (I apologize for including all our remarks but it might make it easier). In my statement, in reality, "Markov" should replace periodic. The arrival time of the pulse will be Markov rather than periodic. That means where it will arrive in time will be heavily biased, but not completely determined. For the nearly periodic pulses, this will be a very big gain over not using the information of the history. That does not mean storing lots and lots of samples, the history means keeping around a few determined parameters. The variance of the arrival times will be one of the parameters determined. I will investigate the rain induced static scenario as well as I had not considered it. I look forward to working this out and hopefully it will add something to this exercise. Bob -----Original Message----- From: owner-linrad@xxxxxxxxxxxxxxxxxxxxxx [mailto:owner-linrad@xxxxxxxxxxxxxxxxxxxxxx]On Behalf Of Leif Åsbrink Sent: Friday, June 27, 2003 14:06 To: linrad@xxxxxxxxxxxxxxxxxxxxxx Subject: RE: [linrad] Linrad Bob, > I am primarily interested in the short duration, > nearly fixed rate pulse trains. I am afraid you will find this is a mistake. A lot of local qrn that can be removed with very high precision is not nearly fixed rate. Obviously static rain is completely random. Powerline noise is generated by a spark gap (bad insulator) that is connected via a capacitor (the good insulators) to a high voltage 50/60Hz line. Each time (100Hz repetition) the voltage goes through zero and the derivative dU/dt is large, there will be a series of pulses with a time interval related to dU/dt, capacitances, ignition and rest voltage of the sparc gap. There is some uncertainty in the voltages that leads to a randomization of the pulse intervals. Powerline noise is pulse trains of maybe a dozen pulses. The pulse trains repeat at 100/120Hz. There is no precise timing. Commutator motor noise, oil burner ignition arcs, most of the qrn sources are not nearly fixed rate. Farmers fences and car ignition noise are the qrn sources with fixed repetition rate. I think one needs a pulse removal algorithm that can handle random distributed pulses well - the distance pulse to pulse will seldomly be exact enough to allow phase information from pulse to pulse anyway. > The longer duration ones the human ear does not > like but seems to tolerate. Maybe, but the longer ones should be removed as well:) > As I have spent 37 years contesting > and about 5 years going to rock concerts, I cannot > tolerate this impulsive noise for long before I am > completely fatigued due to the pain in my inner > ear from the wideband spectrum of these pulses. > So I am motivated! OK. I think you will experience a generally lower noise floor and more weak dx signals as a bonus:) > Fortunately one of my areas of expertise is the > automation of procedures such as the one you > have brought to our attention. What I will be > attempting to do is build on the thing you have > pointed out, and that is you can subtract a > copy of the pulse as it is presented from > the receiver, even though it has been stretched > by the system, since you have found you can > calculate or guess a few parameters and subtract > the pulse sufficiently from the incoming signal. Oooh! There is no guessing at all - the curve fitting is very accurate indeed. > To me this is a math problem. I have a known signal, > a short duration pulse, and it is my job to figure > out three things: > > 1) it happened > 2) what the channel has done to the pulse The calibration procedure is there to set up a filter that will make the channel do exactly what you want to the pulse. This is done with very high accuracy and the pulse that comes out will have 95 to 99% of its energy on a single sample if it happens to be in phase with the sampling clock. It is up to the user to select the pulse shape he wants to have. This is a rather complex matter that involves dynamic range (if 16-bit arithmetics is in use) sampling rate and how anti aliasing is implemented. It is the fact that the pulses are optimally shaped that makes the simple procedure of Linrad reasonably efficient. > 3) remodulate the pulse with the parameters > determined in 2) > 4) subtract it. > > Number 1 and 2 are best done with determining > the parameters of a joint probability distribution. > Since these pulses tend to be nearly periodic, > all one needs to know to tell a lot about where we are > right now in the signal is what we have observed > before. Since these things can wiggle about and > change with time, it is probabilistic in nature > but this "history determines the present" is still > good. This kind of signal or "stochastic process" > is a Markov process, so named after the Russian > mathematician who described their properties. > The process is a pulse has happened or not. I will be extremely interested of the outcome of this approach. For reasons given above I am afraid you will run into difficulties but on the other hand you might be able to remove periodic pulses better - and once that is done it might be easier to get rid of the non-periodic ones. > The observation we get is through a (changing?) > filter plus more gaussian like noise (thermal). This > means that we are not observing this Markov process > directly but through a mask. We can pick out > some features (otherwise Linrad would not work!). But Linrad does not have any "history determines the present" built into it at all. Absolute nothing. Linrad knows exactly what a pulse will look like. It does a curve fitting to extract the parameters because although a pulse is precisely known, its phase relative to the sampling clock is unknown as well as the amplitude and polarisation. Once the parameters are found, the pulse is removed completely. Complications arise when a notch filter is inserted to get rid of some local station. Linrad does not recalculate the pulse response accordingly, something that should not be very time consuming because the notch filters do not have to change rapidly. Linrad uses another strategy, it renormalizes the pulses by taking the bandwidth reduction into account. If 10% of the bandwidth is removed due to notches, the pulse amplitude goes down by 10%. The weak oscillatory structures around the pulse will cange very much more - but that does not matter at all. The correct pulse is subtracted and the error in mis-match of oscillatory behaviour will be on the frequencies where the notches are. They add nothing at the frequency where we look for weak signals:) > This mask makes the system of pulses plus observations > what is called a Hidden Markov Model. > > At the place I work, there was an algorithm discovered > in the early 1960's. It was rediscovered and popularized > by a scientist named Dempster and is known as the > Maximization/ Expectation algorithm. > > I believe this algorithm can be made to directly apply > and that as a result, we can completely automate the > determination of the channel and pulses jointly, and > this will lead to less pulse energy getting passed the > current process as the modeling will be dynamic. This > last piece is a "theorem": That this algorithm will > make the fewest mistakes in the minimize mean square error > (power leakage passed the subtracter). > > I will be writing this software with Frank, AB2KT. > We will write it C and will share it here. Thanks > a lot Leif for pointing out this to us! Those few > issues of QEX with Linrad and the SDR have changed > my hamming quite a bit! I hope we will be able to collect a library of interesting qrm situations where different strategies can be tested. Taking advantage of any currently unused information (e.g. fixed repetition rate) will surely allow an improved accuracy. The main source of difficulties with the Linrad blanker is splatter and keying clicks from strong stations. Wideband noise with quite different characteristics. Fitting a model waveform that includes splatter and keying clics to each strong signal will allow a dramatic improvement on crowded HF bands. It is beyond what I can do in 2 x 100 kHz bandwidth on a Pentium III however;) 73 Leif / SM5BSZLINRADDARNIL