Note: BargeScraper is currently in testing. There may be issues.
What is this:A little script that automatically outputs dreamwidth friendly html for threadtracking
Installation:BargeScraper requires a few things to get up and running. These are the installation steps for windows.
Install Python 3:( Read more... )Configuring python/updating pipOnce the installation is finished open a command prompt (usually done by going to run and entering cmd). Type "python". You should see this:
Type "pip". You should see this:
If for either of these two you do instead see "python is not recognized as an internal or external command.." it means that python (or pip) was not added to the system path. To fix this type "setx PATH PythonPath" into the commandline where PythonPath is the path to where the python.exe file is.
We now have to update pip. Close the commandline and now open it again but this time run as administrator. To open as administrator on windows 8 you can press windows key+x.
more info here. Type "python -m pip install -U pip" and run it.
Install a text editor/setting up foldersWhen it all works we are ready to install a text editor.
I like
notepad++. I install by using the installer and following the instructions.
Now we are getting close. Good job.
Make a new folder for where you want the bargeScraper to live. Mine lives in documents. In the commandline navigate to this folder. The easiest way to do this is to go into the new folder and then copy the path. Then in the commandline type cd and paste the path, then press enter.
Inside this folder on the command line type:
"pip install requests" then press enter
"pip install beautifulsoup4" then press enter
You are now ready.
Configuration and run bargescraper:Open notepad++ (or other text edior), create a new file. In this file copy in the
code from here. Save this file to bargeScrape.py and select save as type python file.
Enter your username and password, which communities you want to scrape (TLV is what is in there) and which months you want to scrape for. Then save again.
Now in the commandline, check that you are in the folder where the file you just made is and then type "python bargeScrape.py". The scraping will take some time. Make yourself some tea.
The html will be outputted to a file in the same folder, you can open this in notepad++ and copy from there.
Configuration optionsCommunities + Coms titles
- These two lists contain the information about the communities you want to scrape. Add urls to the coms list, make sure to add quotation marks (") and separate by commas. I have added the communities for TLV
- Add the title you want to use for each community in the corresponding place in the comsTitle list. If I want to use the title "Daydreams" for the first community I scrape I put "Daydreams" in the first place in the comsTitle list. Remember to add quotation marks and separate by comma
Months
- There are three options for how to say which month you want to scrape. The first is you give it a start month and an optional end month. If you do not give an end month it will include the current month. The format for these months is "YYYY/MM", remember to use quotation marks
- The second option is to add only the specific months you want to scrape. These are good if you want to check say january, march and august, but don't care for the months in between. Add the months you want to scrape to the list months and leave start and enddates as empty strings (""). The format for these months is "YYYY/MM", remember to use quotation marks and seperate months by comma
- The last option is to not give any months, this causes the default behavior which is to scrape only the current month. Leave the months list empty and the start and end date as ""
Other options
-Filename: Change this if you want a different name for the file with the output html
-Conensed: if you set this to True it will put a cut for each month.
-displayName: if the name you log in as is not the same as your display name add your display name here. This can happen if your login name/url has a - and but the username is displayed with an _
-tagsToCheck: if you also want to add a list of the posts where you need to add your tag add the tag you'll be looking for to the list here. You can check for several tags. Remember to use lower letters and surround the tag with quotation marks and separate several tags with comma. You can use spaces here, example "the iron bull"
ThanksThank you to my partner, Claire, for her help and to my beta testers for your feedback, you have helped make this metter
Please share and enjoy the bargescraper and if you need any help you can post a comment below or contact me on @craftyviking on plurk.
dann_0
