Category Archives: Software

Wechselrichter und Stromzähler mit FRITZ!Box verbinden

Es wäre interessant, die Daten der Photovoltaik zur Steuerung von Steckdosen zu nehmen, um Geräte nur bei Überschuss anzuschalten. Und zwar ohne teure extra Hardware oder einen Smartmeter, der auch noch alle Haushaltsdaten nach aussen funkt quasi als Einladung an potentielle Einbrecher …

Die Kopplung geht einfacher als gedacht. Mein Fronius Wechselrichter ist im Heimnetz unter http://192.168.187.1/solar_api/v1/GetPowerFlowRealtimeData.fcgi zu erreichen. Die aktuelle Wattzahl  kommt im json Format.

Die FRITZ!Box kann die Daten zwar im Augenblick nicht verarbeiten (AVM hat dies aber als Produktvorschlag akzeptiert),  dafür kann die FRITZ!Box eine Steckdose schalten. Also brauchen wir doch einen zweiten Rechner zur Überbrückung. Ich nehme dafür einen Raspberry Pi Zero (der Pi Zero 2 ist vergriffen, aber auch nicht notwendig)  am besten in der Ausführung WH mit Pinleiste.  Stromversorgung kommt von einem Netzteil in Nähe des Sicherungskasten/Stromzählers, zum Test über 3 Tage reicht aber auch eine übliche  Powerbank.

Zunächst wird das Pi OS auf eine alte micro SD Karte installiert und die Karte in den Zero gesteckt. Sobald der Zero dann im WLAN erscheint, kann man ihn headless d.h, ohne Peripheriegeräte via SSH weiter konfigurieren.  Als Messeinheit brauchen wir noch einen USB IR Lesekopf, der am Stromzähler das Blinken der Diode ausliest.

Mit einem cronjob werden im Minutenabstand die Daten abgeholt. Die Abfrage des Logarex Wechselstromzählers funktioniert dabei mit

stty -F /dev/ttyUSB0 300 -parodd cs7 -cstopb parenb -ixoff -crtscts -hupcl -ixon -opost -onlcr -isig -icanon -iexten -echo -echoe -echoctl -echok
echo -n -e '\x2F\x3F\x21\x0D\x0A' >> /dev/ttyUSB0
timeout 10s cat /dev/ttyUSB0

Ab einer bestimmten Leistung der PV kann dann die Steckdose eingeschaltet werden, im Prinzip mit

watt=$(curl http://192.168.187.1/solar_api/v1/GetPowerFlowRealtimeData.fcgi | jq '.Body.Data.Inverters[1].P')
if [[ $watt -gt 200 ]]
then
http://fritz.box/webservices/homeautoswitch.lua?switchcmd=setsimpleonoff&ain=13077%200012360-1&onoff=1

Die Hardware passt in ein kleines Gehäuse, die Kosten liegen unter 50€, dazu kommt noch die schaltbare Steckdose.

 

Update 26.1.22

Der Abfragecode ist nun in ein PHP Skript verpackt und gibt die Daten in einer hübschen d3.js Grafik aus.

Im Augenblick läuft die Datensammlung inklusive regelmässiger openweathermap Abfrage um später ein R-CNN zur Vorhersage zu programmieren.

 

Update 23.2.22

Der Lesekopf funktionierte einwandfrei mit dem Drehstromzähler Logarex LK13BDxxxx obwohl die Shell Kommandos nur sehr primitiv in ein PHP Skript eingebaut waren.

Mit dem Einbau eines neuen bidirektionalen Smartmeters SGM-C4-xxxxxx durch unseren Stromversorger funktioniert das  leider nicht mehr. Problem ist nicht etwa, daß die Kommunikation nun mit 9600/8/N/1 läuft, das Problem ist daß die binäre SML Zeichen, die nun jede Sekunde kommen, nicht richtig  interpretiert werden.

stty -F /dev/ttyUSB0 9600 cs8 -cstopb -parenb
timeout 2s cat /dev/ttyUSB0 | od -tx1

Python downgrade für den php_loader machte wenig Sinn, node liess sich auch nicht auf dem Zero installieren und smeterd brach leider mit einer Fehlermeldung ab. Einzig mit libsml hatte ich aber Erfolg

apt-get install uuid-dev uuid-runtime
git clone https://github.com/volkszaehler/libsml
cd libsml
make
examples/sml_server -s /dev/ttyUSB0

Die -s Option ist dabei undokumentiert, aber wichtig denn nur sie findet genau einen Datensatz. Aus Zeilen 1.8.0 und 2.8.0 lässt sich über einen regex Ausdruck der Zählerstand auslesen.

 

Update 2.3.22

Auf Anfrage kann ich auch das komplette Messgerät verleihen, falls jemand denselben Zähler testweise auslesen oder die SD Karte klonen möchte.  Wenn sich der Zero nicht das eigenes WLAN findet, startet er einen eigenen Hotspot, ist damit also autonom überall verwendbar.

Update 6.9.22

Läuft und läuft … Allerdings boote ich nun einmal in 24 Stunden per crontab , da sich manchmal nach 10 Tagen der Zero aufhängt.

0 1 * * * shutdown -r now

Update 8.11.22

Läuft nun seit 10 Monaten ohne einen Ausfall, genauso wie der Gaszähler.


CC-BY-NC

From Markdown to Bookend and back

When writing text in Typora, I need Bookends to insert references.

Export to RTF format for creating the bibliography isn’t a big deal but using the new RTF in Typora needs Pandoc on the command line as Typora’s import function is broken.

brew install pandoc
pandoc --wrap=none --extract-media=images -f rtf -t markdown myfile.rtf -o myfile.md

Seems that pandoc destroys footnotes and inline images making this combination not a first choice for larger documents.


CC-BY-NC

Why I changed my markdown editor from Atom to Typora

I have been customizing Atom over the years but moved recently to Typora  as it has it all built in. Typora will hopefully work also with long documents (which I haven’t tried so far). There is a

  • great focus mode
  • seamless import of figures
  • LaTeX WYSIWYG
  • Flowchart and Mermaid integration
  • and I can insert my Bookend references (via RTF export)


CC-BY-NC

Good scientific graphics

I have lectured about good graphics more than a decade ago – basically introducing Edward Tufte as the “Leonardo da Vinci” of data. With the beginning of 2022  I am now absolutely overwhelmed by the number of excellent tutorials out there. Here is my recommended  list


CC-BY-NC

youth is no guarantee of innovation

I learned only recently about the 2012 James Bond dialogue in the movie Skyfall where he talks to the new and much younger head of Q branch.

Q: Age is no guarantee of efficiency
Bond: And youth is no guarantee of innovation

Nice to see that I recently earned a new Stackoverflow badge while the last SO report says that 90% of all software developers are younger than 45.


CC-BY-NC

The Python nightmare: How to install mask_rcnn and detectron2

Coming from R the most exciting thing with Python is the incompatibility of the different modules…

While I can recommend the Anaconda Navigator, the following commands work only in the terminal.

Thanks to Nouman945 for help with the original mask_rcnn instructions.

conda create --name matterport python=3.6.13 tensorflow==1.15.0 Keras==2.2.4 h5py==2.8.0 pip
conda activate matterport
git clone https://github.com/matterport/Mask_RCNN.git
cd ./Mask_RCNN-master
python -m pip install -r requirements.txt
git clone https://github.com/philferriere/cocoapi.git
python -m pip install pycocotools
python -m pip install git+https://github.com/philferriere/cocoapi.git#subdirectory=PythonAPI
wget https://github.com/matterport/Mask_RCNN/releases/download/v2.1/mask_rcnn_balloon.h5 > Mask_RCNN/mask_rcnn_coco.h5
conda deactivate

For detectron I follow basically detectron2.readthedocs.io

conda create --name detectron2 python=3.6.13 tensorflow==1.15.0 Keras==2.2.4 h5py==2.8.0 pip
conda activate detectron2
conda install pytorch==1.8.0 torchvision==0.9.0 torchaudio==0.8.0 -c pytorch
git clone https://github.com/facebookresearch/detectron2.git
CC=clang CXX=clang++ ARCHFLAGS="-arch x86_64" python -m pip install -e detectron2
conda deactivate

Last but not least the installed apps:


CC-BY-NC

The Apple Keynote Feature we have been waiting for

Apple updated last month Keynote with a feature we have been waiting for years – live video which is super cool to show live internet pages from active polls.

 

insert video only works from the top menu

There is no video source connected at the beginning but can can be added from the right side menu.

Unfortunately the virtual camera of OBS is not recognized but I could easily connect my Blackmagic Atem Mini so I have direct access to any source coming from a second computer running Chrome, Ipad with Sketchbook  and of course  the Z6 HDMI video signal.

 

adding the ATEM Mini from the right menu

 


CC-BY-NC

A comparison of image duplication software

A quick comparison without pretending any objectivity

Open SourceCommercial / Restricted AccessWebUpper Limit ImagesSpeedSensitivitySpecificityComment
Image Verification AssistantNNY1***
ForensicallyNNY1*****
FotoForensicsNNY1****
ImagedupNNY>500*****proof of principle
ImageTwinNYY10*******
LpixelNYY50????
ProofigNYY50?**/***/**
DropletsYNN1***
ImageJYNN1???
SherloqYNN1NANANAtoolbox
figcheckNNY1???scam?
BioVoxxelYNN1???toolbox
max rating is ***

 

I believe that all services should be in the open domain while  bulk or routine data processing could be a business model.

 


CC-BY-NC

When a scientific journal is modifying your images

The Journal of Allergy and Clinical Immunology (JACI) published numerous images that have been heavily modified. For Pubpeer examples see 1, 2, 3, 4, 5, 6, 7

As I learned recently, this happened without the knowledge of the authors. As the duplications are always at the outer edges and originate from the same image, we assume that JACI publishing service inserted some cloned background” to insert”journal style” letter boxes.

Difficult to understand? Here is an example from the first reference.

 

JACI is known to ignore misconduct allegations while they are now slowly responding to post publication peer review.

…the duplications occurred when the publisher’s compositor vendor styled the figure panel label for print. The error did not impact the analysis, results, or conclusions of the article. The original figure appears below. The authors were not responsible for the error. Measures have been taken to prevent reoccurrence of this error for which the publisher accepts full responsibility and apologizes.

So photoshopping is no more allowed at the JACI office in Colorado– but who is accepting this “full responsibility”?

 

29 Aug 2021

@spy_sci pointed  out this morning to a paper in “Cell Journal”  with a similar phenomenon while also @Cheshire has found multiple times over the past year.  @SmutClyde contributed another example from Bioscience Reports

 


CC-BY-NC

False dichotomies

“The false dichotomy between private interest and public good” by https://www.nuffieldbioethics.org/blog

‘Good’ and ‘public benefit’ are subjective concepts and will vary according to individual perceptions and context. Private and public interest are inevitably intertwined and pitting them against each other creates a false dichotomy. For example, if patients cease to trust their clinicians or more broadly the NHS, public good will suffer. Furthermore, extensive exploration of public attitudes towards sharing medical data has found that people approve in general for their data to be used for medical research and for ‘good causes’, whether environmental, social or medical, but they do not approve of their data to be used for commercial purposes or for powerful companies to profit at society’s expense.


CC-BY-NC

Finally I understand JS (a bit)

It produced a lot of headaches. And yes, it has been described already that

as a lot of articles I read about async in JavaScript, especially those that came out around the time ES2015 (ES6) was new, are just plain wrong. I’m going to clear up the falsehood by presenting a real understanding of the way async works in JavaScript.

Some nice examples are at medium.com and freecodecamp as well as a vid at youtube
.

At the end – I have some working code here, that shouldn’t work according to my understanding of concurrency.


CC-BY-NC

How to monitor any website. At any time interval. With Telegram notification. For Free.

These instructions work for OSX Catalina only and are a bit different than the Linux version shown at urlwatch.

python3 -m pip install pyyaml minidb requests keyring appdirs lxml cssselect html2text
python3 -m pip install urlwatch
EDITOR=nano
urlwatch --edit
# CTRL+X leave the editor without saving

Then edit ~/Library/Application Support/urlwatch/urls.yaml

kind: url
max_tries: 3
name: Camptoi
url: https://www.camptoi.de
filter:
- element-by-id: product-desc-18713727425
- html2text:
    body_width: 0
    ignore_images: false
    ignore_links: false
    inline_links: false
    method: pyhtml2text
    pad_tables: false
    single_line_break: true
    unicode_snob: true

After that we create the Telegram bot by sending

/newbot

to user BotFather and record the <API> from the response.

We also need to send a dummy message to the new bot so we can look up the sender <ID>

https://api.telegram.org/bot<API>/getUpdates

<API> and <ID> are both entered into ~/Library/Application Support/urlwatch/urlwatch.yaml

...
  telegram:
    bot_token: '<API>'
    chat_id: '<ID>'
    enabled: true
---

As we have no cron under MacOS we need a parameter file to run launchd. Should be placed at ~/Library/LaunchAgents/com.urlwatch.plist

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
	<dict>
		<key>Label</key>
		<string>com.urlwatch</string>
		<key>Program</key>
		<string>/Users/wjst/opt/anaconda3/bin/urlwatch</string>
		<key>RunAtLoad</key>
        <true/>
        <key>StartInterval</key>
        <integer>9000</integer>
        <key>KeepAlive</key>
         <true/>       
	</dict>
</plist>

Ultimately, the agent is started (or stopped) using

launchctl load -w  ~/Library/LaunchAgents/com.urlwatch.plist               
launchctl unload ~/Library/LaunchAgents/com.urlwatch.plist

CC-BY-NC