Unclassified interesting Tech Material - page 7
https://swlearninglog.blogspot.com/2022/11/unclassified-interesting-tech-material.html
The next chapter for the node heap memory dump saga
problem: the process that generates the heap dump uses about 20-30 times the memory of the dump created and does not return it back to the OS
dump - 2.7 MB; RAM (rss memory)- 51MB
(probably a server without x-server running would be a better machine for running this)
but still memory usage did not come down for several minutes after the dump was written
what methods were tried
- using unix process signal SIGUSR2
- from within the process programmatically
-- tried to handover process control to node (default buffer management)
-- even tried to manage the buffers that were reading and writing the dump
- using a package as explained here: https://medium.com/@wavded/how-to-heap-snapshots-aac9284d5329
pm2 trigger <process-id / process-name> km:heapdump
- if you have a paid pm2 -- lol
surprising that none of the blogs that detailed the process even mentioned this issue - to me that means they have never tried it in the real world
This an important detail, that a developer worth his salt will not miss.
Everyone has mentioned that it takes twice (theoretically I was expecting this too).
but I think the serialization since it is JSON tries to convert the complete object at once - ends up blowing up in memory.
If that is indeed the case -- that the heap memory dump is just a JSON.stringify -- I am judging the ecosystem BIG TIME.
https://github.com/bnoordhuis/node-heapdump/blob/master/src/heapdump.cc
- is a cpp library for reading and writing the heap
- but for a leak like say 1 array with say 20lakh strings - the serialization itself blows the ram requirement out of proportion.
#
Some other blogs on catching memory leaks:
https://blog.risingstack.com/finding-a-memory-leak-in-node-js/
https://github.com/nodejs/help/issues/1484
https://github.com/nodejs/node/issues/22229
https://github.com/nodejs/node/issues/22229 -- this was fixed in 2018 but seems like the issue I am seeing is similar
https://github.com/nodejs/help/issues/1518
https://github.com/nodejs/help/issues/1518#issuecomment-423922086
- good explanation on the thread
https://github.com/nodejs/node/issues/12805
any way to inspect rss contents:
https://stackoverflow.com/questions/71154339/inspect-rss-memory
no -answers yet !! :(
been 9 months
2.
General errors and bugs -
udemy died
template string visible:
3.
Google's understanding of Technical Detail / Detailed Technical Info
Slow clap google !!
Slooooow clap
####### Sunday 20 November 2022 12:43:14 AM IST #######
4.
Correct match of software and hardware will always be after:
https://blogs.sw.siemens.com/embedded-software/2013/02/18/why-c-is-faster-than-assembly/
"hardware aware software, so to speak" will always be faster
####### Sunday 20 November 2022 12:37:46 AM IST #######
5.
My favourite LATEX resource:
All time favourite:
https://en.wikibooks.org/wiki/LaTeX
also, a new entrant:
https://www.overleaf.com/learn/latex/Articles/How_to_change_paragraph_spacing_in_LaTeX
####### Sunday 20 November 2022 12:10:16 AM IST #######
6.
xdotool - special mention
#7 was enabled using the xdotool
https://github.com/jordansissel/xdotool
(may be this one will help me fire-up all required terminals and softwares I need for a project in my preferred window configuration !)
That'd be nice !
may be will publish a gist for this.
####### Sunday 20 November 2022 12:06:31 AM IST #######
7.
Linux - shortcut to print date to an input
https://askubuntu.com/questions/237209/how-to-create-option-to-insert-time-stamp
https://askubuntu.com/a/237210
similar discussion on the linux mint forum
https://forums.linuxmint.com/viewtopic.php?t=238132
bash -c 'sleep 0.3 && xdotool type "####### $(date) #######"'(above is the expression I am using)
the "bash -c" is needed since it is not a system command but rather a bash script that needs to be interpreted.
Now beautiful time stamps will be a part of all blogger entries :)
####### Saturday 19 November 2022 11:52:37 PM IST #######
this helped me on ubuntu 22.04
https://askubuntu.com/a/1379975
- never checked if it was running on x11 or not
running on wayland
ydotool can relpace xdotool for this case
references:
https://github.com/ReimuNotMoe/ydotool/blob/master/manpage/ydotool.1.scd#keyboard-commands
-
-x-x-x-
####### Thursday 30 March 2023 11:22:45 PM IST #######
ydotool is doing better than xdotool !
new expression:
bash -c 'sleep 0.5 && ydotool type --delay 300 --key-delay 30 "####### $(date) #######"'
also, ydotool needs permissions to write to user-input --> and the permission change (basically change in group) needs a restart of the os
8.
Python when to use which async framework:
if io_bound:
if io_very_slow:
print("Use Asyncio")
else:
print("Use Threads")
else:
print("Multi Processing")####### Sunday 20 November 2022 08:01:42 PM IST #######
9.
Diagnosing Memory leaks in nodejs:
https://nodejs.org/en/docs/guides/diagnostics/memory/#debugging
#
- get heap snapshots from early stage of a process (just started) then get a snapshot of the bloated process (once the process has blown the memory limits you are expecting)
- and compare the two
Official 4 ways to get a heap-snapshot
https://nodejs.org/en/docs/guides/diagnostics/memory/using-heap-snapshot/
a leaky example for practice:
https://github.com/naugtur/node-example-heapdump
#
Really worried performance (:P don't use node / javascript)
- get the profiler log to checkout the memory management steps
- allocation, gc, etc
- recording everything with:
https://nodejs.org/en/docs/guides/diagnostics/memory/using-heap-profiler/
- record gc events with:
https://nodejs.org/en/docs/guides/diagnostics/memory/using-gc-traces/
# Other helpful articles:
https://www.valentinog.com/blog/node-usage/
- overview of memory management in the v8 implementation
https://deepu.tech/memory-management-in-v8/
- mark > sweep > compact
https://www.nearform.com/blog/tracking-memory-allocation-node-js/
- overview of heap management in v8
https://v8.dev/blog/trash-talk
- another more detailed explanation of the same 3 step process
- checkout the chapters for details of the topics
10.
XP-pen deco mini 4 - the driver works like a charm on ubuntu
Takes some getting used to, but a very nice product overall.
https://www.xp-pen.com/download-526.html
- needs a system restart before you can start writing.
# cannot use pressure in mouse mode :(
11.
PhpMyAdmin - Handy expression to generate DB backup file name
@DATABASE@_%F_%I-%M-%S-%p
- for formatting string expression guide:
https://www.php.net/manual/en/function.strftime.php
(phpMyAdmin points to strftime as the underlying implementation, such a nice product !)
12.
This does sound like poetry to my ears after years..
https://python-poetry.org/
Like the npm of the python world.
13.
Seem's like he copied Derek Hilderth's resume format and did not credit the source:
https://www.overleaf.com/latex/templates/software-engineer-resume/gqxmqsvsbdjf
- not 100% sure, but very likely that this one was copied.
14.
Developer friendly template for documenting a software library / framework:
https://github.com/amiechen/codrops-scribbler
- includes terminal mimicry for installation and admin commands
- includes a wiki
-
Live demo: https://tympanus.net/Freebies/scribbler/
15.
Gitbooks: https://www.gitbook.com/
Lost all my old books :(
But a good idea to document with markup and store the documentation as a version controlled code-base
16.
Docker overlay2 folder taking large amount of space.
some of the advise works:
https://stackoverflow.com/questions/46672001/is-it-safe-to-clean-docker-overlay2
https://stackoverflow.com/questions/61592088/why-is-docker-filling-up-var-lib-docker-overlay2
- but better to use your head and read the documentation for commands very carefully.
17.
How to get a link to a youtube comment:
https://www.digitional.com/how-to-share-the-link-to-a-specific-comment-in-youtube/
18.
AI-based api attack detection and attack profiling:
https://github.com/metlo-labs/metlo
Coming soon as of 18th Oct 2022; TODO: follow up by Oct 2023
- looks promising
19.
Useful framework for variable naming
This one would be good to hand over to new programmers:
https://betterprogramming.pub/a-useful-framework-for-naming-your-classes-functions-and-variables-e7d186e3189f
20.
Finally !! at-least a procedural explanation on how reCaptcha enterprise (no challenge) works
https://cloud.google.com/recaptcha-enterprise/docs/overview
21.
Crank for Embedded GUI development:
https://support.cranksoftware.com/hc/en-us/articles/360057412931-What-is-Storyboard-
https://www.cranksoftware.com/learn/video-library#product
- well made solution with an abstraction layer that can be auto populated from the output for popular design tools
- the abstraction can them be rendered for multiple architectures (ARM, X86, MIPS, ..) and operating systems (Android, Linux, Linux-RTOS) and rendering libraries (openGL, framebuffer, etc)
- the performance prediction (resource usage etc is also very good feature of the framework, should really come in handy at design and development stage)
22.
HDDs susceptible to vibrations:
https://www.youtube.com/watch?v=tDacjrSCeq4
- the comments are also interesting
(especially some disks would crash because of a song playing at their natural frequencies)
so there were these known disk-song pairs :D
23.
being used by bitnami to give domains to their ip based deployments
https://nip.io/
24.
ECDSA vs RSA
https://sectigostore.com/blog/ecdsa-vs-rsa-everything-you-need-to-know/
25.
Systemd is a blessing and here to stay
https://unix.stackexchange.com/questions/15348/writing-basic-systemd-service-files
I believe almost all the short comings of previous system init systems have been plugged :)
26.
Recently came across the alternates system in debain:
https://man7.org/linux/man-pages/man1/update-alternatives.1.html
- nice framework for having multiple programs installed in parallel and switch between them
27.
Trying to fine something in files(generally a code base) on a server
- my goto grep command
grep -nrwi <file / folder / regex>
prints the file: line number where there was a match with the line.
-A n
and
-B n
flags if you want to checkout n,m lines in the vicinity (after n lines and before m lines ) of match.
28.
Another linux command line hero and friend:
less - reading long files and searching stuff in them perfect tool.
jump to line:
https://stackoverflow.com/a/8586690/2973457
- :<line-no>g (as soon as you press g the buffer will move, so no need to press enter)
- I was not aware of this, been using less for about 10 years now
29.
entering passwords securely on the linux command prompt
from the mariadb documentation:
-pspecifies a password,password. Note that for passwords, unlike the other parameters, there cannot be a space between the the option (-p) and the value (password). It is also not secure to use a password in this way, as other users on the system can see it as part of the command that has been run. If you include the-poption, but leave out the password, you will be prompted for it, which is more secure.
- will add other examples to give a better picture
Eg.
sudo wp --path='/<wp-install-dir>' --prompt=user_pass user update user 30.
Linux check if a user has read / write access to a file:
This is a particularly useful trick
https://unix.stackexchange.com/a/552279
sudo -u otheruser test -w /file/to/test || echo "otheruser cannot write the file"
https://unix.stackexchange.com/a/157673
output all of the permissions in the path in a vertical list.
namei -m /path/to/really/long/directory/with/file/into list all owners and the permissions
namei -l /path/to/really/long/directory/with/file/inhttps://unix.stackexchange.com/questions/82347/how-to-check-if-a-user-can-access-a-given-file
31.
Nice online UUID generator
https://www.uuidgenerator.net/version4
32.
Some more wordpress / php / linux and apache tips
# I have come to understand the wordpress ecosystem policy like this:
5 different ways to achieve the same thing, none of them works completely
:face_palm
# changing a wordpress domain name from say the staging domain to prod.
- change the 'siteurl' and 'home' variables in the cli (this will have no effect on the one in the DB)
:face_palm
- change them in the wp-config: 'WP_HOME' and 'WP_SITEURL'
-- they will reflect in only some of the site behaviour
-- they will not reflect in the site general settings in the admin panel
-- they will not be editable in general settings
-- they will not reflect in the DB
:face_palm
- so now only way is to disable them from wp-config, use a broken UI to modify them in the admin panel settings and then some of the site will work.
Plugins will still not work. (read further to make plugins work)
To make plugins work:
- wordpress stupidly spreads the domain name all around the DB :face_palm
- wordpress cli search replace - only replaces the first occurrence (why did they even make such a half ass-ed tool ??!! that is beyond me)
- so a freemium plugin is required "better search and replace"
https://wordpress.org/plugins/better-search-replace/
(the ecosystem is so broken, that it creates room for such random players, really pathetic)
- now say you want to replace: http://a.b.c.d with https://example.com
- backup the db:
- replace 1
replace: http://a.b.c.d
with: https://example.com
- replace 2
replace the special character escaped version (from plugins like elementor)
replace: http:\/\/43.205.113.162.nip.io
with: https:\/\/onco.com
(matlab ghante ka standardisation nahi hai)
- there will still be some references to the old domain in the guid columns, so far (a few hours of pulbic testing) I have not faces any issue with them.
So I guess let's let them be.
Links for this part:
- using wp-config.php
https://docs.bitnami.com/aws/apps/wordpress/administration/configure-domain/
- wordpress official - none of the ways work completely
https://wordpress.org/support/article/changing-the-site-url/#wp-cli
- some of the symptoms of this issue:
https://wordpress.stackexchange.com/questions/320512/wrong-domain-in-uploads-folder
https://wordpress.org/support/topic/assets-are-loading-from-old-site-after-domain-change/
- this is the most comprehensive guide on this topic:
https://docs.wpvip.com/how-tos/search-replace/
#
correct rewrites for a wordpress installation in a sub-directory:
https://wordpress.org/support/article/htaccess/#basic-wp
https://perishablepress.com/the-htaccess-rules-for-all-wordpress-permalinks/#permalink-rules
- this weird
although a rewrite may come after the alias but in the resolution of a path will get executed before the alias.
caused a lot of confusion
so say for something installed in a folder say /wp-blogs/my-blog1
and the sub-directory path for the website/domain is domain.com/blogs/my-blog1 then the apache section should look like the following: (notice the weird rewrite rule order due the alias being applied after the rewrite)
- a search does not yield this config, had to be figured out by trial and error
Alias "/blogs/my-blog1/" "/wp-blogs/my-blog1/" <Directory "/wp-blogs/my-blog1"> Options -Indexes +FollowSymLinks -MultiViews AllowOverride None Require all granted # BEGIN WordPress <IfModule mod_rewrite.c> RewriteEngine On RewriteRule .* - [E=HTTP_AUTHORIZATION:%{HTTP:Authorization}] RewriteBase /blogs/my-blog1/ RewriteRule ^index\.php$ - [L] RewriteCond %{REQUEST_FILENAME} !-f RewriteCond %{REQUEST_FILENAME} !-d RewriteRule . /blogs/my-blog1/index.php [L] </IfModule> # END WordPress # BEGIN WordPress fix for plugins and themes # Certain WordPress plugins and themes do not properly link to PHP files because of symbolic links # https://github.com/bitnami/bitnami-docker-wordpress-nginx/issues/43 #RewriteEngine On # already switched on above RewriteRule ^bitnami/wordpress-onco-blog-hindi(/.*) $1 [L] # END WordPress fix for plugins and themes # BEGIN nip.io redirection RewriteEngine On RewriteCond %{HTTP_HOST} ^([0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3})(:[0-9]{1,5})?$ RewriteRule ^/?(.*) %{REQUEST_SCHEME}://%1.nip.io%2/$1 [L,R=302,NE] # END nip.io redirection </Directory>
- looking at the config one may think that a path: domain.com/blogs/my-blog1/<blog-slug>
would resolve to path /wp-blogs/my-blog1/<blog-slug> first due to the alias, and things should work,
but without the rewrite block the server throws a 404 error
- only after the rewrite block is added does the blog with slug is found
#
have your wp-config file in a different directory from the rest of the project:
https://www.groovypost.com/howto/improve-wordpress-securitty-wp-config-php-location/
proxy / reverse proxy models are way better in terms of security and accidentally exposing critical files :D but the knickers example was hilarious !
associated risks:
https://wordpress.stackexchange.com/a/74972
#
what does wp-content actually contain:
https://www.elegantthemes.com/blog/tips-tricks/wp-content-a-beginners-guide-to-wordpress-most-important-directory
#
bitnami wordpress image provisioning (disabling say php from starting up):
bitnami documentation for internals is really bad !
now that I am here, it would have been better to setup everything myself :D
(almost always happens when we need to hyper-customize a ready made stack )
- all the provisioning configuration files are in /root ! (took a very long time to figure this one out)
- init script is configured with systemd as the bitnami.service
- the actual script is in /etc/init.d/bitnami
- uses a bitnami inhouse node package for provisioning (provisioner)
located at: sudo nano /opt/bitnami/nami/node_modules/provisioner/lib/provisioner.js
with its cli.js exposed as the external interface to the packages inside
- all the config files for the provisioner are in root as mentioned above
- so to disable say a service like php8.0 and php-fpm the configuration use the following variables
"startAfterInitialize": false,
"skipService": true,
# apache deployment for multiple domains, multiple sub-folder sites:
- suggested multiple vhosts for multiple domains
- for each sub folder in a domain pointing to a different application, use Alias + Directory
- fpm + proxy can be setup as a general way of handling php files all over, or for a specific directory
- want to server the same content on different domains and servers use the ServerAlias to add more domains
good explanation on the ServerAlias parameter
https://serverfault.com/questions/520195/how-does-servername-and-serveralias-work
Eg.
this is an insightful config recipe for proxy + aliases
https://stackoverflow.com/a/5240764/2973457
#
mysql host for user
- localhost (unix socket) vs 127.0.0.1 (tcp/ip loopback) both are treated differently
https://stackoverflow.com/a/10823868/2973457
also the handling varies from php version to version
- also varies from wordpress version to version
first thing to be looked into in case of db connection errors
#
mysql history file
- generally a good idea to clear history after user creation or disable history during that, to protect user passwords.
#
mysql bug - when recreating a deleted user:
https://sebhastian.com/mysql-error-1396/
#
wordpress asks for FTP credentials when installing plugins
- either access to file / folder for the wordpress/php running user failed
- or a setting in the wp-config specifies this behaviour explicitly
define('FS_METHOD', 'direct');
adding above to the config should help, if the permissions are okay#
wordpress cli -
a lot of tasks can be automated
Eg.
$ wp core install --url=wpclidemo.dev --title="WP-CLI" --admin_user=wpcli --admin_password=wpcli --admin_email=info@wp-cli.orgSuccess: WordPress installed successfully.
$ sudo wp --path='/<wp-install-directory>' --prompt=user_pass user update user
- check if wp cli is working for you$ wp --user=wpcli eval 'echo wp_get_current_user()->user_email;' wpcli@example.com
https://developer.wordpress.org/cli/commands/site/create/
Rest of the documentation:
https://make.wordpress.org/cli/handbook/guides/quick-start/
#
wp-config.php --> get auto generated salts
https://api.wordpress.org/secret-key/1.1/salt/
#
downgrade from php8 to php7.4
// Enable PPA for PHP 7.4 in your system and install it.
sudo add-apt-repository ppa:ondrej/php
sudo apt-get update
sudo apt-get install php7.4
sudo apt-get install php7.4-cli php7.4-common php7.4-json php7.4-opcache php7.4-mysql php7.4-mbstring php7.4-zip php7.4-fpm php7.4-intl php7.4-simplexml
// a2dismod disables the php8.0 module by removing those symlinks.
sudo a2dismod php8.0
// a2enmod enables php7.4 module within the apache2 configuration.
sudo a2enmod php7.4
// Restart apache2 service.
sudo service apache2 restart
// Set alternative name path.
sudo update-alternatives --set php /usr/bin/php7.4
sudo update-alternatives --set phar /usr/bin/phar7.4
sudo update-alternatives --set phar.phar /usr/bin/phar.phar7.4
sudo update-alternatives --set phpize /usr/bin/phpize7.4
sudo update-alternatives --set php-config /usr/bin/php-config7.4
something like: $ compgen -c | grep php
inspired by: https://unix.stackexchange.com/a/120825
Ideally it would be nice to have this one work:
sudo update-alternatives --config php
something like nvm / n - is desirable, did not have time to look for it.
#
related to the downgrade above:
get the installed php modules and related info:
https://stackoverflow.com/a/478922/2973457
php -m (list modules)
php -i (list config details)
#
php-fpm binary location
/usr/sbin/
php-fpm socket file location:
/var/run/
#
configuring php-fpm
https://serversforhackers.com/c/php-fpm-configuration-the-listen-directive
with apache as a reverse proxy:
https://cwiki.apache.org/confluence/display/HTTPD/PHPFPMWordpress
https://cwiki.apache.org/confluence/display/HTTPD/PHP-FPM#PHPFPM-unixdomainsocket(UDS)approach
#
php error: mysqli_real_connect(): (HY000/2002): No such file or directory in wp-db.php on line 1753
when connection fails
it is a bit misleading - since you'd expect a port to be not available / user not permitted
this one is looking for a unix.socket !!!
that is why the file not found :facepalm
good discussion this:
https://stackoverflow.com/a/42869490/2973457
#
changing the host for a mariadb user
https://serverfault.com/questions/483339/changing-host-permissions-for-mysql-users
most of them did not work for me.. so ended up creating a new user and copying the privileges (as suggested by phpmyadmin)
TODO: look up more
33.
Quickly encode / decode urls in the browser console:
https://www.w3schools.com/jsref/jsref_encodeuricomponent.asp
34.
React Native allowing Horcruxes and stuff :D
https://github.com/react-native-svg/react-native-svg/issues/1061
- error rendering svg with react native
- you have to allow Horcruxes :D duh !
35.
Log4J
https://www.youtube.com/watch?v=uyq8yxWO1ls
- makes remote code execution possible
36.
Sounds dreamy !
But what if you want to build something unlike any other ? :P
Builder AI
https://www.youtube.com/watch?v=ucy0R65yxPE
- build click through prototypes by just telling the tools a similar product, if available their database
37.
Hosting service review:
Webfaction used to be great with a lot of once l-cick installs and well-managed linux system with multiple users for shared hosting.
The one click installs for git repos, redmine, trac - python, wordpress, general python wsgi applications (django, flask), ruby on rails applications, databases etc were some of the best one click install installations, I have ever seen.
great for hosting static assets as well, the shared might of a large ngnix server for your static assets made sites faster than CDNs serving assets
It seems everyone is annoyed with webfaction selling to godaddy :D
https://news.ycombinator.com/item?id=19377981
some alternatives from the thread:
I will only list out the ultra cheap options.
Ok !
- These guys are truly ultra cheap for large ram requirements:
https://www.ssdnodes.com/pricing/ - 4.25$ per month for 8GB RAM and 160GB HDD
They have a 1-click install apps as well.
https://blog.ssdnodes.com/blog/ssd-nodes-1-click-apps/
- This one is supposedly from the webfaction team
The hosting price is a bit on the higher side.
Installers from the opal team:
https://github.com/opalstack/installers
- for the opal stack only :P generic installers were better
- Seems to be on par with Digital Ocean on the pricing:
https://www.milesweb.in/hosting/vps-hosting/
- Similar pricing to digital ocean:
https://www.a2hosting.com/vps-hosting/unmanaged/compare
- Almost comparable to ssd nodes in pricing for the large servers:
https://www.hostitsmart.com/servers/vps-hosting
- Similar pricing to digital ocean:
https://ideastack.com/
38.
Sendmail limit email sending to a specific domain:
https://serverfault.com/questions/539520/how-to-configure-sendmail-to-only-send-to-specified-domains
- summary: use mailertable
39.
Wordpress multisite migration plugins:
- WP Migrate:
https://deliciousbrains.com/wp-migrate-db-pro/doc/multisite-tools-addon/
-
40.
One of your best friends on a server:
https://linux.die.net/man/1/find
- entry is tribute to how many times the friend has helped out :D
#
some stunts that can be pulled off with find:
- find all files and change their permissionfind /path/to/location -type f -print0|xargs -0 chmod 644
-d for directories
#
exclude directories when find-ing
https://stackoverflow.com/a/4210072/2973457
- exclude all files in a directory (does not step into the directory)
find . -path ./misc -prune -o -name '*.txt' -print
- To exclude multiple directories, OR them between parentheses.
find . -type d \( -path ./dir1 -o -path ./dir2 -o -path ./dir3 \) -prune -o -name '*.txt' -print
41.
AWS - VPC common use-case scenario:
https://docs.aws.amazon.com/vpc/latest/userguide/vpc-scenarios-intro.html
- VPC with a single public subnet
- VPC with public and private subnets (NAT)
- VPC with public and private subnets and AWS Site-to-Site VPN access
- VPC with a private subnet only and AWS Site-to-Site VPN access
42.
Nice tool to expand the CIDR (Classless Inter-Domain Routing) notation:
https://www.ipaddressguide.com/cidr
43.
The selected AMI contains more instance store volumes than the instance allows. Only the first 0 instance store volumes from the AMI will be accessible from the instance
- image does not expect other drives to be mounted
- so the admin will have to:
- create a partition on the disk
- format the partition
- add to fstab for auto mount at boot
- and manually boot this time to check if config is correct and use the drive without a restart
44.
Wordpress and apache tips and tricks:
#
to enable htaccess files:
- AllowOverride must be enabled (good idea to disable it everywhere, for security)
https://phoenixnap.com/kb/how-to-set-up-enable-htaccess-apache
- check if mod_rewrite --> is installed, enabled and loaded
https://stackoverflow.com/questions/869092/how-to-enable-mod-rewrite-for-apache-2-2
- RewriteEngine On
- debugging rewrites:
RewriteLog "logs/rewritelog"
RewriteLogLevel 7
https://stackoverflow.com/a/32429091/2973457
#
wordpress serialised fields:
Eg.
a:2:{s:4:"hash";s:32:"eba575bc6bcb7336afc956557caa2b80";s:8:"newemail";s:26:"<26-chars>";}
#
marking an email as verified directly in the db:
- delete the entry from user_meta that has your new email and hash (warning !! this may leave the email unchanged in places where it is not kept by reference )
Eg.
a:2:{s:4:"hash";s:32:"eba575bc6bcb7336afc956557caa2b80";s:8:"newemail";s:26:"<26-chars>";}
#
changing username in the db directly (not advised; too much to propagate)
https://www.wpbeginner.com/beginners-guide/how-to-change-the-wordpress-admin-email/
- this says there are only 2 places.. but hard to say
#
incomplete quest: disable confirmation email for email change and accept new email directly
https://wordpress.stackexchange.com/questions/177007/how-to-disable-wordpress-confirmation-email-for-new-users
#
user not getting recognized as the network admin / super-admin
mark a user as super-admin:
https://mycyberuniverse.com/wordpress-multisite-network-admin-menu-item-disappeared.html
Edit the site_admins meta key. Change the value to the username to whom you want to give “Super Admin” privileges.
#
cannot add plugins ?
https://www.wpbeginner.com/plugins/why-cant-i-add-or-install-plugins-in-wordpress/
https://wordpress.org/support/topic/can-not-access-network-admin-multisite/
#
routing scheme difference in multisite sub-domain vs sub-directory
- each blog gets a folder in the wordpress content directory - in the file system
- since the admin panel and plugins are shared
the subdomain and the sub folder have different rewrite rules for admin, plugins, content
- they basically eventually point the same file structure with a different starting url
#
folder and db structure difference in single site vs multisite
- table comparison:
https://www.tech-prastish.com/blog/wordpress-multisite-database-structure/
https://getflywheel.com/layout/the-basics-of-wordpress-multisite/
- files: https://multilingualpress.org/docs/wordpress-multisite-overview/#Files
-
#
complete tutorial on multisite:
https://kinsta.com/blog/wordpress-multisite/
#
apache list all available modules:
$ apache2ctl -M OR --------------- On Debian based systems ---------------
$ apache2ctl -t -D DUMP_MODULES --------------- On RHEL based systems --------------- $ apachectl -t -D DUMP_MODULES OR $ httpd -M
https://ubiq.co/tech-blog/check-apache-modules-enabled/
https://www.tecmint.com/check-apache-modules-enabled/
#
Restart required after .htaccess file change ?
- no
- but a hard reload from the browser may be required
https://www.hostgator.com/help/article/must-i-restart-apache-after-changing-a-htaccess-file
#
bitnami documentation for the wordpress ami is not that great but is also not bad :
https://docs.bitnami.com/aws/apps/wordpress-multisite/
https://bitnami.com/stack/wordpress-multisite
- very specific details are not included, you need to find that out from the code.
Eg, where are the rewrite rules added to the config,
- what are the startup scripts
- how does the provisioning work
had to search the entire hierarchy from the httpd.conf right upto the last leaf level, following all the includes :(
#
tunnelling to mysql sometimes does not work for weird reasons
- so better try 127.0.0.1 and localhost both
- for connecting from browser and the tunnel as well -L <local-port>:<127.0.0.1 / server-ip>
#
wordpress multisite vs multiple wordpress sites on a server
https://wordpress.stackexchange.com/a/104754
- pros and cons explained
multiple wordpress
Pros:
Each site lives on it's own and has it's own database
If something breaks with one site it doesn't take down the network
Cons:
Can be a pain to manage/upgrade
Users/accounts can be harder to link as you are dealing with different servers
-x-x-x-
multisite
Pros:
Super easy to mange/upgrade
Everything is in one place
Shared database makes connections between sites (such as customer/user info) easy
Cons:
Can be hard to set up initially
If something breaks all sites could crash
All plugins/themes are linked (although individual access can be controlled)
Some plugins/themes don't work with Multi-Site
On the same topic, architecture guidelines:
https://wordpress.org/support/article/installing-multiple-blogs/
- https://code.tutsplus.com/tutorials/moving-wordpress-moving-a-site-into-a-multisite-network--cms-22773
- https://wpmudev.com/blog/moving-wordpress-site-into-multisite/
https://github.com/10up/MU-Migration
- https://kinsta.com/blog/migrate-wordpress-site/#migrating-a-site-into-or-out-of-wordpress-multisite-with-plugins
Wordpress admin panel cyclic infinite redirect:
- not a unique problem:
https://www.google.com/search?q=wordpress+admin+redirecting+to+login+for+reauth&oq=wordpress+admin+redirecting+to+login+for+reauth&aqs=chrome..69i57.20781j0j7&sourceid=chrome&ie=UTF-8
https://wordpress.org/support/topic/unfixable-redirect-loop-at-wp-login-with-reauth1/
https://core.trac.wordpress.org/ticket/47980
delete cookies and try again
45.
The story of email sending from wordpress installed on an AWS EC2 machine:
#
used the bitnami ami for this
- sendmail / postfix was not installed (facepalm !!)
#
wordpress (send test email from wp mail smtp)
#
so initially even the test mails were failing.
Error: PHPMailer was able to connect to SMTP server but failed while trying to send an email.
- since this was for php based wordpress
- installed sendmail - generally the default for php (no config changes are required generally)
- after installation, no error from php but no email receipt, not even in spam
#
tried sending email from php directly (send email by a sample script)
https://www.php.net/manual/en/function.mail.php
- no error, but no delivery either
#
linux
tried sending email from cli; directly as a command using the sendmail binary
- no error, but not delivery either
#
sendmail logs
/var/log/mail.log- showed that the emails were successfully queued
Message accepted for delivery
- showed that connections to google email servers were timed out
stat=Deferred: Connection timed out with alt4.gmail-smtp-in.l.google.com
#
AWS
- checked VPC
- checked subnet
- checked security group
#
network
- tried to connect to google servers on telnet !!
and then found out that the ports 25, 587, 465 are blocked
and the blocking is implicit - screw you AWS !
please learn from the Zen of python - explicit is better than implicit
Eg.
https://www.linuxquestions.org/questions/linux-networking-3/sendmail-configuration-connection-timed-out-with-alt4-gmail-smtp-in-l-google-com-4175510482/
I am not saying they should not block sensitive ports !!
But be explicit about them in your VPC, subnet, security group settings
Aaaand:
https://aws.amazon.com/premiumsupport/knowledge-center/ec2-windows-email-server-issues/
wrong documentation !! (says, network ACL and security groups to be checked,)
- but checked them, and they were not blocking anything and
Nice !
Finally,
this is the correct info:
https://aws.amazon.com/premiumsupport/knowledge-center/ec2-port-25-throttle/
and the port unblocking is to be requested here:
https://support.console.aws.amazon.com/support/contacts#/rdns-limits
Aaaand when the form finally submits there is no acknowledgement / tracking number displayed..
- redeeming factor: an email is sent to you email on file
46.
AWS EC2 blocks email sending by default
- ye bold mein likhna chahiye salon ne
- had to debug this from highest level in the stack to the lowest to find that the port is blocked..
see #45 for details
47.
Disable starting a service at boot - systemctl
systemctl disable abcd.service
more about it:
https://unix.stackexchange.com/a/516934
48.
Also, recording a terminal session as text and rendering it into a video:
https://asciinema.org/docs/how-it-works
Nicely multiplxed from an existing stream !
49.
Dayum canonical !! Nice !!
Ubuntu VMs on the command line.
Build your own cloud on your machine.
https://multipass.run/install
50.
Comparing Mocha, Jest, Jasmine:
https://dev.to/heroku/comparing-the-top-3-javascript-testing-frameworks-2cco
- great approach to explaining the strengths and weaknesses of each of the frameworks
51.
Some rclone tips
- google oauth flow - need to complete on browser
- be careful with the steps, any one missed step will lead to not very well documented errors from google
- if the drive-shared-with-me flag is set --> only shared with me folder is shown in ls
AND it also affects sync - when it lists files (meaning clone doesn’t see the existing folder and creates a duplicate)
https://forum.rclone.org/t/sync-to-google-drive-multiple-duplicate-folders/32798/2
https://forum.rclone.org/t/how-to-prevent-creating-duplicate-folders-in-google-drive-with-server-side-transfer/7278/3
Docs are great:
Installation:
https://rclone.org/install/
(with go based tools, I prefer to use pre-built binaries)
Config:
for google drive remote:
https://rclone.org/drive/#configuration
- where to find the config file:
https://rclone.org/docs/#config-config-file
52.
Omg uploading a lot of small files to google drive is such a pain and so slow.
go for tar / tar.gz
best results so far:
rclone sync --progress --verbose --transfers=10 --tpslimit=10 --bwlimit 9M <local-source> <google-drive-dest>53.
Ubuntu extra ttys with ctrl + alt + f[1-7] key combinations:
Been using these ttys for about 10 years now, not sure if these are available on other distro flavours as well
'''
In fact there are not only 6 or 7 ttys, there are many more. You could see them, try
ls /dev/tty* | wc -l
All the consoles are not always active. You need to activate ttyN in order to switch to it with Ctrl+Alt+F N. You can activate any tty with openvt command also. See man openvt.
'''
https://askubuntu.com/a/385875/547592
54.
I get that it is a security feature:
https://git-scm.com/docs/git-config/2.35.2#Documentation/git-config.txt-safedirectory
- but there should be a way to disable it.
- does not support regular expressions
- it is annoying
55.
Get a backup of your gitlab repos from all projects:
https://gitlab.com/-/snippets/2406387
- python script
- requirements.txt included
- using a virtual env. is recommended
tools used:
- gitlab python api - https://python-gitlab.readthedocs.io/en/stable/api-usage.html
to crawl the contents of all your gitlab projects
- pyyaml - https://pyyaml.org/wiki/PyYAMLDocumentation
to generate a yaml for vcs to use
- vcstool - https://github.com/dirk-thomas/vcstool
to clone all the repos at once
-x-x-x-x-
####### Sunday 14 May 2023 08:49:59 PM IST #######
when establishing identity is an issue with vcs cloning
- advice:
- use ssh
- in ssh config create a custom host
- and convert all the git@gitlab.com --> gitlab_personal in the repo ssh address
Host gitlab_personal HostName gitlab.com User git IdentitiesOnly yes IdentityFile ~/.ssh/<path-to-rsa-file-private>/id_rsa
- discussion about this in a general context:
https://stackoverflow.com/questions/4565700/how-to-specify-the-private-ssh-key-to-use-when-executing-shell-command-on-git
-x-x-x-
some of the other tools that I chose not to use, mostly due to lack of cloning from a config file feature:
https://myrepos.branchable.com/
https://stackoverflow.com/questions/3497123/run-git-pull-over-all-subdirectories
https://medium.com/@codenameyau/updating-multiple-repos-with-one-command-9768c8cdfe46
https://dzone.com/articles/dealing-with-multiple-git-repositories
- git sub-modules: https://git-scm.com/book/en/v2/Git-Tools-Submodules
56.
Nx for scaling javascript projects:
https://www.youtube.com/watch?v=VUyBY72mwrQ
- first class support for most common javascript frameworks, react, angular, express, nest
- dependency aware change detection --> only affected parts are built are tested
- testing integrated with Jest and Cypress
57.
Nice essay advocating mono-repo.
https://monorepo.tools/
built by:
https://nx.dev/getting-started/intro
Been using this format for 3 years. (2019-2022)
Have had no problems with lerna
Found this to be most impressive:
https://v1.pantsbuild.org/why_use_pants.html
supports: javascript, java, python
pants internals: https://www.youtube.com/watch?v=IL6LBWNi3fE
Why use monorepos ? and why not ?
https://www.youtube.com/watch?v=VvcJGjjEyKo (this guy makes some valid points)
https://www.youtube.com/watch?v=lV8-1S28ycM Uber (multi --> mono and back to multi again)
58.
Google drive linux clients:
https://linuxhint.com/best_google_drive_clients_linux/
1. Insync
2. Rclone - cli --> seems like the most mature option as of now.
trying this out.
TODO: review
3. OverGrive
4. GoSync - python, gui, early stage
5. Grive2 - fork from Grive
6. CloudCross - complete cloud data movement platform
7. Gdrive - cli tool
8. Google-drive-ocamlfuse - let's you mount google-drive as a drive
9. DriveSync - ruby, cli
10. GNOME Online Accounts
11. Google drive sync:
https://github.com/dtsvetkov1/Google-Drive-sync
59.
Very cool data moving services, very good business indeed.
(started with Google Drive to Dropbox moving services)
https://www.movebot.io/pricing?hsLang=en
https://www.multcloud.com/price
https://inclowdz.wondershare.com/store/individuals.html
60.
I have seen the depths:
https://www.youtube.com/watch?v=pEfrdAtAmqk
- very nicely put together language classifications in terms of depth of knowledge required to really really understand and work with this current ecosystem of computers
61.
LUA: faster and simpler than python: Interesting way to be fast:
https://www.youtube.com/watch?v=jUuqBZwwkQw
62.
LLVM is magic:
LLVM short summary:
https://www.youtube.com/watch?v=BT2Cv-Tjq7Q
63.
Busybox: A very useful bundle
https://busybox.net/FAQ.html#whatis
64.
Scylla DB - for
https://www.scylladb.com/open-source-nosql-database/
- project seems to have gained a lot of recognition in the past few years
65.
Kibana index lifecycle management -
https://www.elastic.co/guide/en/elasticsearch/reference/7.17/overview-index-lifecycle-management.html#overview-index-lifecycle-management
- great transition apis defined
- looks very much like a state machine
- and behaviour is very similar to a logger (rotation by file size limit)
66.
A good talk on meta programming:
https://www.youtube.com/watch?v=SrKj4hYic5A
Big question now is:
- when to stop abstracting ?
A great answer I herd somewhere, don't remember where...
When the returns to effort ratio <= 1
67.
Getting a good view of the disk usage
https://www.redhat.com/sysadmin/sort-du-output
related to #69. but this is a CLI only approach.
#
du --all --human-readable --one-file-system --max-depth=1 /var
#
du -ahx --max-depth=1 /var
#
du -ahx --max-depth=1 /var | sort -k1 -rh
68.
Configure remote desktop server ubuntu:
https://www.howtogeek.com/429190/how-to-set-up-remote-desktop-on-ubuntu/
69.
Tools to get disk usage reports and visualisations:
https://www.makeuseof.com/tag/how-to-analyze-your-disk-usage-pattern-in-linux/
- df and du combination (very useful on servers)
- ncdu
- Qdirstat
- disk usage analyser (a.k.a baobab what the heck is wrong with the name ?!)
- xdiskusage
- duc
- jDiskReport
- why no caching baobab ?
currently no way no cache / way to store output in a file ?
https://bugs.launchpad.net/ubuntu/+source/baobab/+bug/942255
70.
A good command for system backup
rsync --exclude="sys" --exclude="proc" --exclude="bin" --exclude="boot" --exclude="dev" --exclude="lib" --exclude="lib64" --exclude="lost"+found --exclude="media" --exclude="mnt" --exclude="opt" --exclude="run" --exclude="sbin" --exclude="snap" --exclude="srv" --exclude="tmp" --exclude="usr" -arv --partial --progress --rsh=ssh <root-user-preferably>@<ip-or-domain>:/ /<destination>
71.
How to verify that the ssh key you uploaded to gitlab is working correctly ?
ssh -T git@gitlab.example.comhttps://docs.gitlab.com/ee/user/ssh.html#verify-that-you-can-connect
72.
Ever realised how critical this piece of software is to the success of linux servers :) ?
https://ubuntu.com/server/docs/service-openssh
73.
Get all the users for a linux system
multiple ways to do it:
cut -d: -f1 /etc/passwd
sed 's/:.*//' /etc/passwd
awk -F: '{print $1}' /etc/passwd
grep -oE '^[^:]+' /etc/passwd
https://unix.stackexchange.com/a/65266
74.
Ubuntu / Linux: Mounting an external usb drive, HDD
also, a check can be performed with lsusb to check if the HDD is communicating well with the usb hardware on the motherboard.
(older HDDs need more power that newer USB ports cannot provide)
sudo fdisk -l
mount /dev/sdb1 /mnt
(different distros and versions of ubuntu have a separate preference for the mounting directory)
Eg. /media/<username>/<drive-name>
/mnt/<drive-name>
etc...
Not necessary to follow any convention, it can be decided on case by case basis
-x-x-x-
this one is particularly useful for dealing with HDDs plugged into a machine that you are ssh-ing into.
75.
Trouble shooting sound issues on ubuntu
- issues:
-- digital noise in the sound output (through all the output options available - 3.5mm, HDMI etc)
-- options disappearing from sound settings on selecting any other configuration than stereo output
- step 1 worked for me
https://help.ubuntu.com/community/SoundTroubleshootingProcedure
76.
Get the response body in an express middleware:
- Eg. say to calculate the length
https://thewebdev.info/2022/03/06/how-to-log-the-response-body-with-express/
this is how you do you it.
const logResponseBody = (req, res, next) => { const oldWrite = res.write const oldEnd = res.end; const chunks = []; res.write = (chunk, ...args) => { chunks.push(chunk); return oldWrite.apply(res, [chunk, ...args]); }; res.end = (chunk, ...args) => { if (chunk) { chunks.push(chunk); } const body = Buffer.concat(chunks).toString('utf8'); console.log(req.path, body); return oldEnd.apply(res, [chunk, ...args]); }; next(); } app.use(logResponseBody);
- this chaining is very stupid in Javascript.
- I think people need to move away from express.
77.
Did not know that the Arduino project was based on these two:
IDE - https://processing.org/
SDK - http://wiring.org.co/
78.
Gitlab - why the transfer process so convoluted ?
Transfer gitlab repo from one user to another
https://gitlab.com/gitlab-org/gitlab/-/issues/14502
Currently, the only way to transfer a project from one user to another is:
- Create a dummy group for which both users are owners
- Have the old user transfer ownership of the project from themselves to the group
- Have the new user transfer ownership from the group to themselves
- Delete the dummy group
- how to transfer a gitlab repo to a new namespace:
https://docs.gitlab.com/ee/user/project/settings/
79.
Elastic search / Open search - shard management
- sometimes shards seem to be unnecessarily allocated to indices
- seems like there is no way to reduce the shards for an index once it is created, which is weird
To get the current distribution of shards:
https://www.elastic.co/guide/en/elasticsearch/reference/current/cat-shards.html
To see why the particular allocation for a shard:
https://www.elastic.co/guide/en/elasticsearch/reference/current/cluster-allocation-explain.html
Shrink index gives out a new index:
https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-shrink-index.html
- Only way to delete shards seems to be to delete indices.
- It seems that indices where one shard is dedicated for replication to other nodes.. only those shards can be deleted by reducing the replication factor for the index
Usually happens automatically, but can help reduce the space an index and shards taking by removing deleted / duplicated entries between shards:
https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-forcemerge.html
Thankfully not dealing with a situation as complicated as this:
https://thoughts.t37.net/how-to-fix-your-elasticsearch-cluster-stuck-in-initializing-shards-mode-ce196e20ba95
- since there are no replication nodes for this setup yet
This is good guide for reducing shards:
https://opster.com/guides/elasticsearch/capacity-planning/elasticsearch-reduce-shards/
if only unassigned shards is your problem:
https://www.datadoghq.com/blog/elasticsearch-unassigned-shards/
-x-x-x-x-
my solution to the problem
# get shards, nodes, indices list GET _cat/shards/_all/?h=index,shard,node,p,state,unassigned.reason,docs,store # get the data inside the index to make sure no critical data is being deleted # below is just an example GET /.kibana_-1200753886_<username> GET /.kibana_-1200753886_<username>/_search { "query": { "match_all": {} }, "size": 20 } Executed commands # delete the unwanted auto-generated indices DELETE /.kibana_-1200753886_<username>_1
80.
Install ghost on ubuntu:
Official guide:
https://ghost.org/docs/install/ubuntu/#run-the-install-process
From fast comet:
https://www.fastcomet.com/tutorials/ghost/manual-ghost-installation
Ghost CLI: (good tools for most part of the setup and maintenance)
https://ghost.org/docs/ghost-cli/#ghost-setup
The config options are also pretty exhaustive:
https://ghost.org/docs/config/#admin-url
This is the weird part of the installation:
https://www.fastcomet.com/tutorials/ghost/create-admin-user
- creating the admin user happens on the gui 🤦🤦🤦🤦
81.
Digital ocean firewalls:
https://docs.digitalocean.com/products/networking/firewalls/how-to/create/
- similar to aws
both creation and config:
https://docs.digitalocean.com/products/networking/firewalls/how-to/configure-rules/
82.
Javascript debugging tips and tricks:
https://www.youtube.com/watch?v=_QtUGdaCb1c
- console.dir
- console.table
- console.time and console.timeEnd
these were new and nice.
Debugging is not console.log ! :)
But console object instance is powerful for sure
83.
Blogger advanced search options:
Blogger otherwise is pathetic, this was a surprise
https://support.google.com/blogger/answer/9675453?hl=en&authuser=0&visit_id=637900334009281442-2812545331&p=search-operators&rd=1
84.
How to change Ghost password for admin user in the db
https://www.geekinsta.com/how-to-reset-ghost-password/
To reset the password of your website:
- Login to the server using SSH.
- Connect to the MySQL database using
mysql -u username -p. Enter the password if necessary. Replaceusernamewith the database username. - Execute the following query to reset the password.
UPDATE your_database_here.users SET password='hashed_password' WHERE email = 'login_email'; - Login as Ghost admin.
Or there is some other issue.
85.
Bcrypt hash generator and verifier:
https://bcrypt.online/
Nice tool.
86.
System wide Equalizer for linux:
- Pulse audio multiband EQ
- pulse effects - I am not sure if it same as above --
87.
Random name pickers:
- useful tools for picking random names for a draw:
https://www.dcode.fr/random-selection
https://commentpicker.com/random-name-picker.php
88.
Why gnome why ? - simple options for desktop wallpaper settings moved to tweaks
https://askubuntu.com/a/1023090/547592
this is stupid.
I'll fix it if I get a chance to look into it
89.
Recommended Javascript courses - free and otherwise
most recommended on top
https://javascript.info/
- covers all the basics + promises and async / await
https://javascript.info/ebook
- would love to buy and keep it as a basic guide for new comers
This is a good list of free courses:
- one can easily pick one and get started
https://www.freecodecamp.org/news/learn-javascript-free-js-courses-for-beginners/
Some free udemy courses:
- Decent course - the content is a little lacking in all round completeness
https://www.udemy.com/course/javascript-essentials/
- Good coverage - only async / await and promises seem missing
https://www.udemy.com/course/programming-in-javascript/
- covers almost everything (minus async await)
https://www.udemy.com/course/javascriptfundamentals/
90.
Docker Extensions framework:
https://www.docker.com/blog/build-your-first-docker-extension/
-- Kudos docker team; nicely done
Thought of everything:
development, hot reload, debugging
91.
Alter mouse speed - linux
- did not find much about it
TODO: figure this out, of-course after I am annoyed out of my mind
92.
Shout out to the best git + npm based mono-repo management tool !!
https://github.com/lerna/lerna#readme
-- been using it for 2 years now and am yet to find something that is not yet available with the package.
93.
Django DB Backup:
Some things in the python-django ecosystem are soooo sooo sorted:
https://django-dbbackup.readthedocs.io/en/master/configuration.html
- get back-up from most major dbs
- can backup all project data (media files too)
- can back-up to multiple supported destinations - s3, google drive, dropbox, etc
- clean-up with plugable custom function to determine deletion
- clean-up with last 'n' backups also supported
- cron / celery to schedule regular backups
and many more desirable features
Honestly !! Aur kua chahiye ?
94.
How to get the PostgreSQL version that is running server / client
https://chartio.com/resources/tutorials/how-to-view-which-postgres-version-is-running/
95.
PgAdmin !! Amazingly well written software
especially with version 4 !
Perfect for admin tasks for a postgres db.
https://www.pgadmin.org/download/
https://www.pgadmin.org/download/pgadmin-4-apt/
96.
Screw you Django !! Why such cryptic errors ?
What does a key error during migrations mean ?
Case 1:
KeyError('<django-app>', '<model>')
$ python manage.py makemigrations Traceback (most recent call last): File "manage.py", line 22, in <module> main() File "manage.py", line 18, in main execute_from_command_line(sys.argv) File "/home/ubuntu/venv/lib/python3.8/site-packages/django/core/management/__init__.py", line 419, in execute_from_command_line utility.execute() File "/home/ubuntu/venv/lib/python3.8/site-packages/django/core/management/__init__.py", line 413, in execute self.fetch_command(subcommand).run_from_argv(self.argv) File "/home/ubuntu/venv/lib/python3.8/site-packages/django/core/management/base.py", line 354, in run_from_argv self.execute(*args, **cmd_options) File "/home/ubuntu/venv/lib/python3.8/site-packages/django/core/management/base.py", line 398, in execute output = self.handle(*args, **options) File "/home/ubuntu/venv/lib/python3.8/site-packages/django/core/management/base.py", line 89, in wrapped res = handle_func(*args, **kwargs) File "/home/ubuntu/venv/lib/python3.8/site-packages/django/core/management/commands/makemigrations.py", line 149, in handle loader.project_state(), File "/home/ubuntu/venv/lib/python3.8/site-packages/django/db/migrations/loader.py", line 335, in project_state return self.graph.make_state(nodes=nodes, at_end=at_end, real_apps=list(self.unmigrated_apps)) File "/home/ubuntu/venv/lib/python3.8/site-packages/django/db/migrations/graph.py", line 315, in make_state project_state = self.nodes[node].mutate_state(project_state, preserve=False) File "/home/ubuntu/venv/lib/python3.8/site-packages/django/db/migrations/migration.py", line 89, in mutate_state operation.state_forwards(self.app_label, new_state) File "/home/ubuntu/venv/lib/python3.8/site-packages/django/db/migrations/operations/fields.py", line 92, in state_forwards state.models[app_label, self.model_name_lower].fields[self.name] = field KeyError: ('insurance', 'networkhospital')
Case 2:
KeyError('<model-field>')
$ python manage.py makemigrations Traceback (most recent call last): File "manage.py", line 22, in <module> main() File "manage.py", line 18, in main execute_from_command_line(sys.argv) File "/home/ubuntu/venv/lib/python3.8/site-packages/django/core/management/__init__.py", line 419, in execute_from_command_line utility.execute() File "/home/ubuntu/venv/lib/python3.8/site-packages/django/core/management/__init__.py", line 413, in execute self.fetch_command(subcommand).run_from_argv(self.argv) File "/home/ubuntu/venv/lib/python3.8/site-packages/django/core/management/base.py", line 354, in run_from_argv self.execute(*args, **cmd_options) File "/home/ubuntu/venv/lib/python3.8/site-packages/django/core/management/base.py", line 398, in execute output = self.handle(*args, **options) File "/home/ubuntu/venv/lib/python3.8/site-packages/django/core/management/base.py", line 89, in wrapped res = handle_func(*args, **kwargs) File "/home/ubuntu/venv/lib/python3.8/site-packages/django/core/management/commands/makemigrations.py", line 149, in handle loader.project_state(), File "/home/ubuntu/venv/lib/python3.8/site-packages/django/db/migrations/loader.py", line 335, in project_state return self.graph.make_state(nodes=nodes, at_end=at_end, real_apps=list(self.unmigrated_apps)) File "/home/ubuntu/venv/lib/python3.8/site-packages/django/db/migrations/graph.py", line 315, in make_state project_state = self.nodes[node].mutate_state(project_state, preserve=False) File "/home/ubuntu/venv/lib/python3.8/site-packages/django/db/migrations/migration.py", line 89, in mutate_state operation.state_forwards(self.app_label, new_state) File "/home/ubuntu/venv/lib/python3.8/site-packages/django/db/migrations/operations/fields.py", line 162, in state_forwards old_field = model_state.fields.pop(self.name) KeyError: '<model-field>'
Everything wrong with this:
- if you don't know that all models and fields are turned to small case by django for db keys, no way in hell are you going to relate the keys printed here to your apps, models and fields
- if you do not realize that the strings there are actually from the models and fields and names of your apps then tough knuckles
- no mention of the file where the error actually is
got a hint from a zero upvoted answer:
https://stackoverflow.com/a/46224403
nobody is talking about the fact that this is such a misleading screwed up error and tells the developer nothing about what really went wrong, one cannot even get a hint on where to look.
- Trawling a lot of forums and QnA about the topic makes one realise that these error mean that the migrations did not find your app, or model,
- case 1 ( KeyError('<django-app>', '<model>') ):
the model in the app is missing or has not been created yet;
(finally I realised that means, the model was not created in any of the migrations before the one that is using it)
got a hit on the model grepping all the migrations files available in all of the apps.
and realized that there is no mention of "creating" the model that is being modified in this migration file....
So added a CreateModel with appropriate params.. so that the later migration can in principle modify it.
- case 2
KeyError('<model-field>')
a migration is trying to modify a model-field before it has been declared.
solution:
- add the field in the migration which has the CreateModel for the model
- add a filed to the model with AddField before it being modified (just has to be in one of the migrations before the problematic one)
TODO:
- answer the issue on stack-overflow (the answers to the questions there are sparsely populated / non-existent)
https://stackoverflow.com/questions/67591021/keyerror-while-migrating-in-python
https://stackoverflow.com/questions/30242243/keyerror-profiles-talk-how-do-i-resolve/46224403
- file this with the django team that manages migrations
97.
Doc strings in Javascript
https://dev.to/stephencweiss/write-your-own-javascript-contracts-and-docstrings-42ho
- what should you capture according to me
-- input
-- output
-- any special thing the function takes care of that is not obvious from the name
What not to write:
-- general walk through of how the function does what it does (becomes difficult to maintain across code changes)
-- I like commented code blocks for that
98.
Redis Commander:
- great software !!
- used to manage redis and visually inspect key-values-ttl
99.
Multi application database:
Nice ecosystem these guys have built:
https://www.snowflake.com/
https://www.snaplogic.com/blog/snowflake-data-platform
100.
Getting battery percentage info from a Bluetooth device providing the information:
- I prefer the local install version:
https://github.com/TheWeirdDev/Bluetooth_Headset_Battery_Level/tree/v1.3.1#option-2-download-this-repository
cd Bluetooth_Headset_Battery_Level
chmod +x bluetooth_battery.py
./bluetooth_battery.py BT_MAC_ADDRESS_1 ...make sure you have python-pybluez or python3-pybluez or python3-bluez installed on your system.
https://stackoverflow.com/a/59709851/2973457
https://www.reddit.com/r/linux/comments/glcani/check_battery_level_of_your_bluetooth_headset_in/
https://askubuntu.com/a/1223990/547592
https://answers.launchpad.net/ubuntu/+question/698455
many places pointed to the same project.
Did not try this one..
This is also mentioned in multiple places:
https://askubuntu.com/a/1207887/547592
Continued from:
https://swlearninglog.blogspot.com/2022/01/unclassified-interesting-tech-material.html
Comments
Post a Comment